Sep 08 00:33:06.197258 ip-10-0-139-137 systemd[1]: Starting Kubernetes Kubelet... Sep 08 00:33:06.558433 ip-10-0-139-137 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 00:33:06.558433 ip-10-0-139-137 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 08 00:33:06.558433 ip-10-0-139-137 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 00:33:06.558433 ip-10-0-139-137 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 08 00:33:06.558433 ip-10-0-139-137 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 00:33:06.560838 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.560699 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 08 00:33:06.564557 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564543 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 00:33:06.564557 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564558 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 00:33:06.564621 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564562 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 00:33:06.564621 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564565 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 00:33:06.564621 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564569 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 00:33:06.564621 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564571 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 00:33:06.564621 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564574 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 00:33:06.564621 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564577 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 00:33:06.564621 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564580 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 00:33:06.564621 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564590 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 00:33:06.564621 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564594 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 00:33:06.564621 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564598 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 00:33:06.564621 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564602 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 00:33:06.564621 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564604 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 00:33:06.564621 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564607 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 00:33:06.564621 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564610 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 00:33:06.564621 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564612 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 00:33:06.564621 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564615 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 00:33:06.564621 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564617 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 00:33:06.564621 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564620 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 00:33:06.564621 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564623 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 08 00:33:06.565117 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564625 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 00:33:06.565117 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564628 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 00:33:06.565117 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564632 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 00:33:06.565117 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564635 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 00:33:06.565117 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564639 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 00:33:06.565117 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564642 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 00:33:06.565117 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564645 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 00:33:06.565117 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564648 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 00:33:06.565117 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564651 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 00:33:06.565117 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564654 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 00:33:06.565117 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564657 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 00:33:06.565117 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564660 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 00:33:06.565117 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564662 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 00:33:06.565117 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564664 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 00:33:06.565117 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564667 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 00:33:06.565117 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564670 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 00:33:06.565117 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564672 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 00:33:06.565117 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564674 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 00:33:06.565117 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564677 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 00:33:06.565837 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564679 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 00:33:06.565837 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564682 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 00:33:06.565837 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564684 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 00:33:06.565837 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564687 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 00:33:06.565837 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564691 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 00:33:06.565837 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564695 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 00:33:06.565837 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564698 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 00:33:06.565837 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564700 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 00:33:06.565837 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564703 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 00:33:06.565837 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564705 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 00:33:06.565837 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564708 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 00:33:06.565837 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564710 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 00:33:06.565837 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564713 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 00:33:06.565837 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564715 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 00:33:06.565837 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564719 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 00:33:06.565837 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564722 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 00:33:06.565837 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564725 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 00:33:06.565837 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564727 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 00:33:06.565837 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564730 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 00:33:06.565837 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564732 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 00:33:06.566439 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564735 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 00:33:06.566439 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564737 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 00:33:06.566439 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564740 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 00:33:06.566439 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564743 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 00:33:06.566439 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564745 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 00:33:06.566439 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564747 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 00:33:06.566439 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564750 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 00:33:06.566439 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564752 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 00:33:06.566439 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564755 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 00:33:06.566439 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564757 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 00:33:06.566439 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564760 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 00:33:06.566439 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564762 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 00:33:06.566439 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564765 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 00:33:06.566439 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564768 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 00:33:06.566439 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564770 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 00:33:06.566439 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564772 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 00:33:06.566439 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564775 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 00:33:06.566439 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564778 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 00:33:06.566439 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564781 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 00:33:06.566439 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564783 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 00:33:06.566987 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564786 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 00:33:06.566987 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564788 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 00:33:06.566987 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564791 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 00:33:06.566987 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564793 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 00:33:06.566987 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564797 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 00:33:06.566987 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564802 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 00:33:06.566987 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564805 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 00:33:06.566987 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564808 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 00:33:06.566987 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564811 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 00:33:06.566987 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564814 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 00:33:06.566987 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564816 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 00:33:06.566987 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564819 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 00:33:06.566987 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564822 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 00:33:06.566987 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564824 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 00:33:06.566987 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564827 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 00:33:06.566987 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564829 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 00:33:06.566987 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564832 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 00:33:06.566987 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564835 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 00:33:06.566987 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.564837 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 00:33:06.567930 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.567914 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 00:33:06.567930 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.567931 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 00:33:06.567989 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.567935 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 00:33:06.567989 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.567938 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 00:33:06.567989 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.567941 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 00:33:06.567989 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.567944 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 00:33:06.567989 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.567947 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 00:33:06.567989 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.567950 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 00:33:06.567989 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.567952 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 00:33:06.567989 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.567955 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 00:33:06.567989 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.567958 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 00:33:06.567989 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.567961 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 00:33:06.567989 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.567963 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 00:33:06.567989 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.567966 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 00:33:06.567989 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.567969 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 00:33:06.567989 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.567971 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 00:33:06.567989 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.567974 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 00:33:06.567989 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.567976 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 00:33:06.567989 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.567979 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 00:33:06.567989 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.567983 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 00:33:06.567989 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.567986 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 00:33:06.568500 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.567989 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 00:33:06.568500 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568006 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 00:33:06.568500 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568009 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 00:33:06.568500 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568012 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 00:33:06.568500 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568015 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 00:33:06.568500 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568018 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 00:33:06.568500 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568021 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 00:33:06.568500 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568024 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 00:33:06.568500 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568026 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 00:33:06.568500 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568029 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 00:33:06.568500 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568032 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 00:33:06.568500 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568035 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 00:33:06.568500 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568037 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 00:33:06.568500 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568040 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 00:33:06.568500 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568042 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 00:33:06.568500 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568045 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 00:33:06.568500 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568047 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 08 00:33:06.568500 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568050 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 00:33:06.568500 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568052 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 00:33:06.568965 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568055 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 00:33:06.568965 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568057 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 00:33:06.568965 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568059 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 00:33:06.568965 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568062 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 00:33:06.568965 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568066 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 00:33:06.568965 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568069 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 00:33:06.568965 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568072 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 00:33:06.568965 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568075 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 00:33:06.568965 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568078 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 00:33:06.568965 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568080 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 00:33:06.568965 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568083 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 00:33:06.568965 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568085 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 00:33:06.568965 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568088 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 00:33:06.568965 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568091 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 00:33:06.568965 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568094 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 00:33:06.568965 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568096 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 00:33:06.568965 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568099 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 00:33:06.568965 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568101 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 00:33:06.568965 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568104 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 00:33:06.569477 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568107 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 00:33:06.569477 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568109 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 00:33:06.569477 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568111 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 00:33:06.569477 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568114 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 00:33:06.569477 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568117 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 00:33:06.569477 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568119 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 00:33:06.569477 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568122 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 00:33:06.569477 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568125 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 00:33:06.569477 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568127 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 00:33:06.569477 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568130 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 00:33:06.569477 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568132 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 00:33:06.569477 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568135 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 00:33:06.569477 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568137 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 00:33:06.569477 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568140 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 00:33:06.569477 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568142 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 00:33:06.569477 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568145 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 00:33:06.569477 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568147 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 00:33:06.569477 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568149 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 00:33:06.569477 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568152 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 00:33:06.569477 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568156 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 00:33:06.569961 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568159 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 00:33:06.569961 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568161 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 00:33:06.569961 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568164 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 00:33:06.569961 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568166 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 00:33:06.569961 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568170 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 00:33:06.569961 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568172 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 00:33:06.569961 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568175 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 00:33:06.569961 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568178 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 00:33:06.569961 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568181 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 00:33:06.569961 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568184 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 00:33:06.569961 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568186 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 00:33:06.569961 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568188 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 00:33:06.569961 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568191 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 00:33:06.569961 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568193 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 00:33:06.569961 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568196 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 00:33:06.569961 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568198 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 00:33:06.569961 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568200 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 00:33:06.569961 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568203 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 00:33:06.569961 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568205 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 00:33:06.569961 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568208 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 00:33:06.570468 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568618 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 00:33:06.570468 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568624 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 00:33:06.570468 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568629 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 00:33:06.570468 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568632 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 00:33:06.570468 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568635 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 00:33:06.570468 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568638 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 00:33:06.570468 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568641 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 00:33:06.570468 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568644 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 00:33:06.570468 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568646 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 00:33:06.570468 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568649 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 00:33:06.570468 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568652 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 00:33:06.570468 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568655 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 00:33:06.570468 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568657 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 00:33:06.570468 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568660 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 00:33:06.570468 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568663 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 00:33:06.570468 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568665 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 00:33:06.570468 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568668 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 00:33:06.570468 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568672 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 00:33:06.570468 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568674 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 00:33:06.570931 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568677 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 00:33:06.570931 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568680 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 00:33:06.570931 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568682 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 00:33:06.570931 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568685 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 00:33:06.570931 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568688 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 00:33:06.570931 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568690 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 00:33:06.570931 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568692 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 00:33:06.570931 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568695 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 00:33:06.570931 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568698 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 00:33:06.570931 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568700 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 00:33:06.570931 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568702 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 00:33:06.570931 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568705 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 00:33:06.570931 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568708 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 00:33:06.570931 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568710 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 00:33:06.570931 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568713 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 00:33:06.570931 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568715 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 00:33:06.570931 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568718 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 00:33:06.570931 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568720 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 00:33:06.570931 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568723 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 00:33:06.570931 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568725 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 00:33:06.571434 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568727 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 00:33:06.571434 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568730 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 00:33:06.571434 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568733 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 00:33:06.571434 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568735 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 00:33:06.571434 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568737 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 00:33:06.571434 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568741 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 00:33:06.571434 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568745 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 00:33:06.571434 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568748 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 00:33:06.571434 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568750 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 00:33:06.571434 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568753 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 00:33:06.571434 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568756 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 00:33:06.571434 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568759 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 00:33:06.571434 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568761 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 00:33:06.571434 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568764 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 00:33:06.571434 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568766 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 00:33:06.571434 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568769 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 00:33:06.571434 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568773 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 00:33:06.571434 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568777 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 00:33:06.571434 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568779 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 00:33:06.571900 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568782 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 00:33:06.571900 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568785 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 00:33:06.571900 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568787 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 00:33:06.571900 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568789 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 00:33:06.571900 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568792 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 00:33:06.571900 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568795 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 00:33:06.571900 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568797 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 00:33:06.571900 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568799 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 00:33:06.571900 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568802 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 00:33:06.571900 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568804 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 00:33:06.571900 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568807 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 00:33:06.571900 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568809 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 00:33:06.571900 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568812 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 00:33:06.571900 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568814 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 00:33:06.571900 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568817 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 00:33:06.571900 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568819 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 00:33:06.571900 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568821 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 00:33:06.571900 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568824 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 00:33:06.571900 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568826 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 00:33:06.571900 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568829 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 00:33:06.572412 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568832 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 00:33:06.572412 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568834 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 00:33:06.572412 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568837 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 00:33:06.572412 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568839 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 00:33:06.572412 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568843 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 00:33:06.572412 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568846 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 00:33:06.572412 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568848 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 00:33:06.572412 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568850 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 00:33:06.572412 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568853 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 00:33:06.572412 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568856 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 00:33:06.572412 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568859 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 00:33:06.572412 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568861 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 08 00:33:06.572412 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568864 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 00:33:06.572412 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568866 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 00:33:06.572412 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568869 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 00:33:06.572412 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568871 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 00:33:06.572412 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568874 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 00:33:06.572412 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568876 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 00:33:06.572412 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568878 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 00:33:06.572412 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568881 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.568883 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.568956 2577 flags.go:64] FLAG: --address="0.0.0.0" Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.568989 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569009 2577 flags.go:64] FLAG: --anonymous-auth="true" Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569014 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569018 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569021 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569025 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569028 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569031 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569034 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569038 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569041 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569044 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569047 2577 flags.go:64] FLAG: --cgroup-root="" Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569050 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569053 2577 flags.go:64] FLAG: --client-ca-file="" Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569057 2577 flags.go:64] FLAG: --cloud-provider="external" Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569060 2577 flags.go:64] FLAG: --cluster-dns="[]" Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569064 2577 flags.go:64] FLAG: --cluster-domain="" Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569067 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569069 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569072 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 08 00:33:06.572902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569076 2577 flags.go:64] FLAG: --container-log-max-files="5" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569080 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569082 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569085 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569088 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569091 2577 flags.go:64] FLAG: --contention-profiling="false" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569094 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569097 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569100 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569103 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569107 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569111 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569114 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569117 2577 flags.go:64] FLAG: --enable-load-reader="false" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569120 2577 flags.go:64] FLAG: --enable-server="true" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569122 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569127 2577 flags.go:64] FLAG: --event-burst="100" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569130 2577 flags.go:64] FLAG: --event-qps="50" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569133 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569136 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569139 2577 flags.go:64] FLAG: --eviction-hard="" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569142 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569145 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569148 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569151 2577 flags.go:64] FLAG: --eviction-soft="" Sep 08 00:33:06.573512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569154 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569157 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569160 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569163 2577 flags.go:64] FLAG: --experimental-mounter-path="" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569166 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569169 2577 flags.go:64] FLAG: --fail-swap-on="true" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569172 2577 flags.go:64] FLAG: --feature-gates="" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569175 2577 flags.go:64] FLAG: --file-check-frequency="20s" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569178 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569181 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569184 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569187 2577 flags.go:64] FLAG: --healthz-port="10248" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569190 2577 flags.go:64] FLAG: --help="false" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569193 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-139-137.ec2.internal" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569196 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569199 2577 flags.go:64] FLAG: --http-check-frequency="20s" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569202 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569205 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569208 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569211 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569214 2577 flags.go:64] FLAG: --image-service-endpoint="" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569217 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569219 2577 flags.go:64] FLAG: --kube-api-burst="100" Sep 08 00:33:06.574137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569222 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569225 2577 flags.go:64] FLAG: --kube-api-qps="50" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569228 2577 flags.go:64] FLAG: --kube-reserved="" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569231 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569236 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569239 2577 flags.go:64] FLAG: --kubelet-cgroups="" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569242 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569245 2577 flags.go:64] FLAG: --lock-file="" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569248 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569250 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569253 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569259 2577 flags.go:64] FLAG: --log-json-split-stream="false" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569262 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569265 2577 flags.go:64] FLAG: --log-text-split-stream="false" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569268 2577 flags.go:64] FLAG: --logging-format="text" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569271 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569274 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569276 2577 flags.go:64] FLAG: --manifest-url="" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569279 2577 flags.go:64] FLAG: --manifest-url-header="" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569284 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569286 2577 flags.go:64] FLAG: --max-open-files="1000000" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569290 2577 flags.go:64] FLAG: --max-pods="110" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569293 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569296 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569299 2577 flags.go:64] FLAG: --memory-manager-policy="None" Sep 08 00:33:06.574764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569302 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569306 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569309 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569312 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569319 2577 flags.go:64] FLAG: --node-status-max-images="50" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569322 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569325 2577 flags.go:64] FLAG: --oom-score-adj="-999" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569328 2577 flags.go:64] FLAG: --pod-cidr="" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569331 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569337 2577 flags.go:64] FLAG: --pod-manifest-path="" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569340 2577 flags.go:64] FLAG: --pod-max-pids="-1" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569343 2577 flags.go:64] FLAG: --pods-per-core="0" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569347 2577 flags.go:64] FLAG: --port="10250" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569350 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569353 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bbffc35b1d65177b" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569356 2577 flags.go:64] FLAG: --qos-reserved="" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569359 2577 flags.go:64] FLAG: --read-only-port="10255" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569362 2577 flags.go:64] FLAG: --register-node="true" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569365 2577 flags.go:64] FLAG: --register-with-taints="" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569368 2577 flags.go:64] FLAG: --registry-burst="10" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569372 2577 flags.go:64] FLAG: --registry-qps="5" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569375 2577 flags.go:64] FLAG: --reserved-cpus="" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569377 2577 flags.go:64] FLAG: --reserved-memory="" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569381 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569384 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 08 00:33:06.575378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569387 2577 flags.go:64] FLAG: --rotate-certificates="false" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569390 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569392 2577 flags.go:64] FLAG: --runonce="false" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569395 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569398 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569401 2577 flags.go:64] FLAG: --seccomp-default="false" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569404 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569406 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569410 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569413 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569416 2577 flags.go:64] FLAG: --storage-driver-password="root" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569419 2577 flags.go:64] FLAG: --storage-driver-secure="false" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569422 2577 flags.go:64] FLAG: --storage-driver-table="stats" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569424 2577 flags.go:64] FLAG: --storage-driver-user="root" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569427 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569430 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569433 2577 flags.go:64] FLAG: --system-cgroups="" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569436 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569441 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569445 2577 flags.go:64] FLAG: --tls-cert-file="" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569448 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569451 2577 flags.go:64] FLAG: --tls-min-version="" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569454 2577 flags.go:64] FLAG: --tls-private-key-file="" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569457 2577 flags.go:64] FLAG: --topology-manager-policy="none" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569459 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 08 00:33:06.575986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569462 2577 flags.go:64] FLAG: --topology-manager-scope="container" Sep 08 00:33:06.576962 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569465 2577 flags.go:64] FLAG: --v="2" Sep 08 00:33:06.576962 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569469 2577 flags.go:64] FLAG: --version="false" Sep 08 00:33:06.576962 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569473 2577 flags.go:64] FLAG: --vmodule="" Sep 08 00:33:06.576962 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569476 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 08 00:33:06.576962 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.569480 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 08 00:33:06.576962 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569591 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 00:33:06.576962 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569596 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 00:33:06.576962 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569599 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 00:33:06.576962 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569601 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 00:33:06.576962 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569604 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 00:33:06.576962 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569607 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 00:33:06.576962 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569610 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 00:33:06.576962 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569612 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 00:33:06.576962 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569615 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 00:33:06.576962 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569619 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 00:33:06.576962 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569622 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 00:33:06.576962 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569625 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 00:33:06.576962 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569627 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 00:33:06.576962 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569630 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 00:33:06.576962 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569633 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 00:33:06.576962 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569635 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 00:33:06.577890 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569638 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 00:33:06.577890 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569640 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 00:33:06.577890 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569643 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 00:33:06.577890 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569645 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 00:33:06.577890 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569650 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 00:33:06.577890 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569652 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 00:33:06.577890 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569655 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 00:33:06.577890 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569659 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 00:33:06.577890 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569663 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 00:33:06.577890 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569666 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 00:33:06.577890 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569669 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 00:33:06.577890 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569672 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 00:33:06.577890 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569675 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 00:33:06.577890 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569678 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 00:33:06.577890 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569681 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 00:33:06.577890 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569684 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 00:33:06.577890 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569686 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 00:33:06.577890 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569689 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 00:33:06.577890 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569691 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 00:33:06.578731 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569694 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 00:33:06.578731 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569696 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 00:33:06.578731 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569699 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 00:33:06.578731 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569702 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 00:33:06.578731 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569704 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 00:33:06.578731 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569707 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 00:33:06.578731 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569710 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 00:33:06.578731 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569713 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 00:33:06.578731 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569715 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 00:33:06.578731 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569718 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 08 00:33:06.578731 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569720 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 00:33:06.578731 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569723 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 00:33:06.578731 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569726 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 00:33:06.578731 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569728 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 00:33:06.578731 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569730 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 00:33:06.578731 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569733 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 00:33:06.578731 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569735 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 00:33:06.578731 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569739 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 00:33:06.578731 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569742 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 00:33:06.578731 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569744 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 00:33:06.579729 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569747 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 00:33:06.579729 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569749 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 00:33:06.579729 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569752 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 00:33:06.579729 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569755 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 00:33:06.579729 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569757 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 00:33:06.579729 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569759 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 00:33:06.579729 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569762 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 00:33:06.579729 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569764 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 00:33:06.579729 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569767 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 00:33:06.579729 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569770 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 00:33:06.579729 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569773 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 00:33:06.579729 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569775 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 00:33:06.579729 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569778 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 00:33:06.579729 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569781 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 00:33:06.579729 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569783 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 00:33:06.579729 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569786 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 00:33:06.579729 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569788 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 00:33:06.579729 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569791 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 00:33:06.579729 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569795 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 00:33:06.579729 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569797 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 00:33:06.580405 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569800 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 00:33:06.580405 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569802 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 00:33:06.580405 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569805 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 00:33:06.580405 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569807 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 00:33:06.580405 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569810 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 00:33:06.580405 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569812 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 00:33:06.580405 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569815 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 00:33:06.580405 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569817 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 00:33:06.580405 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569820 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 00:33:06.580405 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569825 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 00:33:06.580405 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569829 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 00:33:06.580405 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569832 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 00:33:06.580405 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569834 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 00:33:06.580405 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569837 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 00:33:06.580405 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569840 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 00:33:06.580405 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569842 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 00:33:06.580405 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569845 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 00:33:06.580405 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569847 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 00:33:06.580405 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569850 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 00:33:06.580405 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569852 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 00:33:06.581113 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569855 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 00:33:06.581113 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569858 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 00:33:06.581113 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569862 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 00:33:06.581113 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.569865 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 00:33:06.581113 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.570290 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 00:33:06.581113 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.578415 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 08 00:33:06.581113 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.578436 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 08 00:33:06.581113 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.578467 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 08 00:33:06.581113 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.578477 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 08 00:33:06.581113 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578564 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 00:33:06.581113 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578573 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 00:33:06.581113 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578578 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 00:33:06.581113 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578583 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 00:33:06.581113 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578588 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 00:33:06.581705 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578593 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 00:33:06.581705 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578598 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 00:33:06.581705 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578602 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 00:33:06.581705 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578607 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 00:33:06.581705 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578611 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 00:33:06.581705 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578615 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 00:33:06.581705 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578620 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 00:33:06.581705 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578624 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 00:33:06.581705 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578628 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 00:33:06.581705 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578633 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 00:33:06.581705 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578637 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 00:33:06.581705 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578641 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 00:33:06.581705 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578645 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 00:33:06.581705 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578649 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 00:33:06.581705 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578653 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 00:33:06.581705 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578658 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 00:33:06.581705 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578662 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 00:33:06.581705 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578666 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 00:33:06.581705 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578670 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 00:33:06.581705 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578674 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 00:33:06.582446 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578681 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 00:33:06.582446 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578686 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 00:33:06.582446 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578691 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 00:33:06.582446 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578696 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 00:33:06.582446 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578703 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 00:33:06.582446 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578708 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 00:33:06.582446 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578713 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 00:33:06.582446 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578717 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 00:33:06.582446 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578721 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 00:33:06.582446 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578725 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 00:33:06.582446 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578729 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 00:33:06.582446 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578733 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 00:33:06.582446 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578737 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 00:33:06.582446 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578741 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 00:33:06.582446 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578744 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 00:33:06.582446 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578749 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 00:33:06.582446 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578752 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 00:33:06.582446 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578759 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 00:33:06.582446 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578765 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 00:33:06.582968 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578770 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 00:33:06.582968 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578775 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 00:33:06.582968 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578780 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 00:33:06.582968 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578785 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 00:33:06.582968 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578789 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 00:33:06.582968 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578794 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 00:33:06.582968 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578798 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 00:33:06.582968 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578803 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 00:33:06.582968 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578807 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 00:33:06.582968 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578812 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 00:33:06.582968 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578816 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 00:33:06.582968 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578821 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 00:33:06.582968 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578826 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 00:33:06.582968 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578830 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 00:33:06.582968 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578834 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 00:33:06.582968 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578838 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 00:33:06.582968 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578843 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 00:33:06.582968 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578848 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 00:33:06.582968 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578853 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 00:33:06.582968 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578857 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 00:33:06.583561 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578862 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 00:33:06.583561 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578866 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 00:33:06.583561 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578872 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 00:33:06.583561 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578877 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 00:33:06.583561 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578882 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 00:33:06.583561 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578886 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 00:33:06.583561 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578890 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 00:33:06.583561 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578894 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 00:33:06.583561 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578898 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 00:33:06.583561 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578903 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 00:33:06.583561 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578907 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 00:33:06.583561 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578911 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 00:33:06.583561 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578915 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 00:33:06.583561 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578919 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 00:33:06.583561 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578923 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 00:33:06.583561 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578927 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 00:33:06.583561 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578931 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 00:33:06.583561 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578935 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 00:33:06.583561 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578939 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 00:33:06.583561 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578943 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 08 00:33:06.584126 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578947 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 00:33:06.584126 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578951 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 00:33:06.584126 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578956 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 00:33:06.584126 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578960 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 00:33:06.584126 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578964 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 00:33:06.584126 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578968 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 00:33:06.584126 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578972 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 00:33:06.584126 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578977 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 00:33:06.584126 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578981 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 00:33:06.584126 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578986 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 00:33:06.584126 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.578990 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 00:33:06.584126 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579011 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 00:33:06.584126 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579015 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 00:33:06.584126 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579020 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 00:33:06.584126 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579024 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 00:33:06.584538 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.579032 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 00:33:06.584538 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579215 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 00:33:06.584538 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579224 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 00:33:06.584538 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579229 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 00:33:06.584538 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579233 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 00:33:06.584538 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579238 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 00:33:06.584538 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579242 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 00:33:06.584538 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579247 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 00:33:06.584538 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579251 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 00:33:06.584538 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579256 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 00:33:06.584538 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579260 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 00:33:06.584538 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579264 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 00:33:06.584538 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579269 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 00:33:06.584538 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579273 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 00:33:06.584538 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579277 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 00:33:06.584970 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579281 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 00:33:06.584970 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579285 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 00:33:06.584970 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579289 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 00:33:06.584970 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579293 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 00:33:06.584970 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579297 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 00:33:06.584970 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579303 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 00:33:06.584970 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579311 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 00:33:06.584970 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579316 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 00:33:06.584970 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579320 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 00:33:06.584970 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579326 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 00:33:06.584970 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579332 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 00:33:06.584970 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579338 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 00:33:06.584970 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579342 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 00:33:06.584970 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579346 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 00:33:06.584970 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579352 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 00:33:06.584970 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579356 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 00:33:06.584970 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579361 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 00:33:06.584970 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579365 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 00:33:06.584970 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579369 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 00:33:06.585492 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579373 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 00:33:06.585492 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579378 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 00:33:06.585492 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579381 2577 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 00:33:06.585492 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579385 2577 feature_gate.go:328] unrecognized feature gate: Example Sep 08 00:33:06.585492 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579389 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 00:33:06.585492 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579393 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 00:33:06.585492 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579397 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 00:33:06.585492 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579401 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 00:33:06.585492 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579405 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 00:33:06.585492 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579408 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 00:33:06.585492 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579412 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 00:33:06.585492 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579416 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 00:33:06.585492 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579419 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 00:33:06.585492 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579423 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 00:33:06.585492 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579428 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 00:33:06.585492 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579432 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 00:33:06.585492 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579436 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 00:33:06.585492 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579440 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 00:33:06.585492 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579444 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 00:33:06.585492 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579448 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 00:33:06.586009 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579452 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 00:33:06.586009 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579456 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 00:33:06.586009 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579460 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 00:33:06.586009 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579464 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 00:33:06.586009 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579468 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 00:33:06.586009 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579473 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 00:33:06.586009 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579477 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 00:33:06.586009 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579481 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 00:33:06.586009 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579485 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 00:33:06.586009 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579490 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 00:33:06.586009 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579494 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 00:33:06.586009 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579499 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 00:33:06.586009 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579503 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 00:33:06.586009 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579507 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 00:33:06.586009 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579511 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 00:33:06.586009 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579516 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 00:33:06.586009 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579520 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 00:33:06.586009 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579524 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 00:33:06.586009 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579530 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 00:33:06.586788 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579536 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 00:33:06.586788 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579542 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 00:33:06.586788 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579546 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 00:33:06.586788 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579551 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 00:33:06.586788 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579556 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 00:33:06.586788 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579561 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 00:33:06.586788 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579565 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 00:33:06.586788 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579569 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 00:33:06.586788 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579574 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 00:33:06.586788 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579578 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 00:33:06.586788 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579583 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 00:33:06.586788 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579587 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 00:33:06.586788 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579591 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 00:33:06.586788 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579595 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 00:33:06.586788 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579599 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 00:33:06.586788 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579603 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 00:33:06.586788 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579607 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 00:33:06.586788 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579612 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 00:33:06.586788 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579617 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 00:33:06.586788 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579621 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 00:33:06.587796 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579625 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 00:33:06.587796 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579629 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 00:33:06.587796 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579635 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 00:33:06.587796 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579639 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 00:33:06.587796 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579643 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 00:33:06.587796 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579647 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 00:33:06.587796 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:06.579651 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 00:33:06.587796 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.579659 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 00:33:06.587796 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.579828 2577 server.go:962] "Client rotation is on, will bootstrap in background" Sep 08 00:33:06.587796 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.582946 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 08 00:33:06.587796 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.583868 2577 server.go:1019] "Starting client certificate rotation" Sep 08 00:33:06.587796 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.583969 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 00:33:06.587796 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.584010 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 00:33:06.588377 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.586667 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 00:33:06.588918 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.588895 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 00:33:06.603164 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.603145 2577 log.go:25] "Validated CRI v1 runtime API" Sep 08 00:33:06.606143 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.606127 2577 log.go:25] "Validated CRI v1 image API" Sep 08 00:33:06.607035 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.607021 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 08 00:33:06.609906 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.609884 2577 fs.go:135] Filesystem UUIDs: map[38d5aa24-210e-4e3c-afd2-ecb1233b350c:/dev/nvme0n1p4 44b3b6f0-d52d-4b76-8f9c-391068472074:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Sep 08 00:33:06.609967 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.609906 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 08 00:33:06.614920 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.614809 2577 manager.go:216] Machine: {Timestamp:2026-09-08 00:33:06.614309385 +0000 UTC m=+0.309550535 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3078750 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24bd3bdffefdf3df50c562a46d03ee SystemUUID:ec24bd3b-dffe-fdf3-df50-c562a46d03ee BootID:5f800d57-36a0-4008-b1a5-184f2145ff2c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d7:cf:d6:c6:f1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d7:cf:d6:c6:f1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:72:93:26:d2:bd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 08 00:33:06.614920 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.614914 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 08 00:33:06.615047 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.615008 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 08 00:33:06.616701 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.616676 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 08 00:33:06.616844 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.616702 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-137.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 08 00:33:06.617482 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.617470 2577 topology_manager.go:138] "Creating topology manager with none policy" Sep 08 00:33:06.617541 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.617486 2577 container_manager_linux.go:309] "Creating device plugin manager" Sep 08 00:33:06.617541 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.617494 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 00:33:06.618145 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.618135 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 00:33:06.619808 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.619798 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 08 00:33:06.620070 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.620051 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 00:33:06.623348 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.623336 2577 state_mem.go:36] "Initialized new in-memory state store" Sep 08 00:33:06.623463 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.623454 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 08 00:33:06.623499 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.623485 2577 kubelet.go:486] "Attempting to sync node with API server" Sep 08 00:33:06.623499 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.623497 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 08 00:33:06.623552 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.623509 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 08 00:33:06.623552 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.623521 2577 kubelet.go:398] "Adding apiserver pod source" Sep 08 00:33:06.623552 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.623530 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 08 00:33:06.623919 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.623906 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 00:33:06.623979 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.623923 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 00:33:06.626298 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.626278 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 08 00:33:06.630969 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.630952 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 08 00:33:06.631075 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.630991 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 08 00:33:06.631211 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.631198 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 08 00:33:06.631268 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.631217 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 08 00:33:06.631268 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.631225 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 08 00:33:06.631268 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.631234 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 08 00:33:06.631268 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.631246 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 08 00:33:06.631268 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.631255 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 08 00:33:06.631759 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.631746 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 08 00:33:06.631819 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.631760 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 08 00:33:06.631819 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.631771 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 08 00:33:06.631819 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.631779 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 08 00:33:06.631819 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.631792 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 08 00:33:06.631819 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.631804 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 08 00:33:06.632465 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.632454 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 08 00:33:06.632525 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.632467 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 08 00:33:06.633618 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:06.633596 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 08 00:33:06.633921 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:06.633900 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-137.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 08 00:33:06.636398 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.636371 2577 server.go:1268] "Started kubelet" Sep 08 00:33:06.636535 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.636479 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 08 00:33:06.636605 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.636561 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 08 00:33:06.636701 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.636672 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 08 00:33:06.637418 ip-10-0-139-137 systemd[1]: Started Kubernetes Kubelet. Sep 08 00:33:06.637614 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.637492 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-137.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 08 00:33:06.638201 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.638184 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 08 00:33:06.639410 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.638967 2577 server.go:310] "Adding debug handlers to kubelet server" Sep 08 00:33:06.642734 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:06.641782 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-137.ec2.internal.18d3313661e0df1f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-137.ec2.internal,UID:ip-10-0-139-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-137.ec2.internal,},FirstTimestamp:2026-09-08 00:33:06.636341023 +0000 UTC m=+0.331582172,LastTimestamp:2026-09-08 00:33:06.636341023 +0000 UTC m=+0.331582172,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-137.ec2.internal,}" Sep 08 00:33:06.644182 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:06.644148 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 08 00:33:06.644359 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.644339 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 08 00:33:06.644896 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.644883 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 08 00:33:06.645012 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:06.644976 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-137.ec2.internal\" not found" Sep 08 00:33:06.645625 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.645601 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 08 00:33:06.645625 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.645600 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 08 00:33:06.645625 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.645628 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 08 00:33:06.645784 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.645773 2577 reconstruct.go:97] "Volume reconstruction finished" Sep 08 00:33:06.645824 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.645785 2577 reconciler.go:29] "Reconciler: start to sync state" Sep 08 00:33:06.646839 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.646824 2577 factory.go:55] Registering systemd factory Sep 08 00:33:06.646839 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.646840 2577 factory.go:223] Registration of the systemd container factory successfully Sep 08 00:33:06.647071 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.647059 2577 factory.go:153] Registering CRI-O factory Sep 08 00:33:06.647120 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.647075 2577 factory.go:223] Registration of the crio container factory successfully Sep 08 00:33:06.647152 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.647141 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 08 00:33:06.647185 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.647162 2577 factory.go:103] Registering Raw factory Sep 08 00:33:06.647185 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.647177 2577 manager.go:1196] Started watching for new ooms in manager Sep 08 00:33:06.647585 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.647571 2577 manager.go:349] Starting recovery of all containers Sep 08 00:33:06.649247 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:06.649220 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 08 00:33:06.649247 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:06.649220 2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-137.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 08 00:33:06.654589 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.654575 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7lg4f" Sep 08 00:33:06.658659 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.658637 2577 manager.go:354] Recovery completed Sep 08 00:33:06.663308 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.663181 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 00:33:06.665439 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.665415 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-137.ec2.internal" event="NodeHasSufficientMemory" Sep 08 00:33:06.665509 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.665453 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-137.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 00:33:06.665509 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.665467 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-137.ec2.internal" event="NodeHasSufficientPID" Sep 08 00:33:06.666011 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.665982 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 08 00:33:06.666011 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.666009 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 08 00:33:06.666162 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.666019 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 00:33:06.666162 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.666028 2577 state_mem.go:36] "Initialized new in-memory state store" Sep 08 00:33:06.666657 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.666645 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-137.ec2.internal" event="NodeHasSufficientMemory" Sep 08 00:33:06.666709 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.666667 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-137.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 00:33:06.666709 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.666677 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-137.ec2.internal" event="NodeHasSufficientPID" Sep 08 00:33:06.667453 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.667438 2577 policy_none.go:49] "None policy: Start" Sep 08 00:33:06.667533 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.667457 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 08 00:33:06.667533 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.667471 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 08 00:33:06.667977 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:06.667908 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-137.ec2.internal.18d33136639cdbc4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-137.ec2.internal,UID:ip-10-0-139-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-137.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-137.ec2.internal,},FirstTimestamp:2026-09-08 00:33:06.665438148 +0000 UTC m=+0.360679304,LastTimestamp:2026-09-08 00:33:06.665438148 +0000 UTC m=+0.360679304,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-137.ec2.internal,}" Sep 08 00:33:06.668645 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.668633 2577 policy_none.go:47] "Start" Sep 08 00:33:06.669502 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.669485 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7lg4f" Sep 08 00:33:06.692623 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.692608 2577 manager.go:337] "Starting Device Plugin manager" Sep 08 00:33:06.692707 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:06.692647 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 08 00:33:06.692707 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.692661 2577 server.go:87] "Starting device plugin registration server" Sep 08 00:33:06.692890 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.692876 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 08 00:33:06.692950 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.692894 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 08 00:33:06.693150 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.693126 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 08 00:33:06.693376 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.693361 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 08 00:33:06.693468 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.693378 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 08 00:33:06.693651 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:06.693636 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 08 00:33:06.693714 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:06.693687 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-137.ec2.internal\" not found" Sep 08 00:33:06.782935 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.782761 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 08 00:33:06.784244 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.784224 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 08 00:33:06.784354 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.784283 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 08 00:33:06.784354 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.784310 2577 kubelet.go:2448] "Starting kubelet main sync loop" Sep 08 00:33:06.784465 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:06.784419 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 08 00:33:06.792159 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.792138 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 00:33:06.793112 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.793098 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 00:33:06.793914 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.793901 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-137.ec2.internal" event="NodeHasSufficientMemory" Sep 08 00:33:06.794007 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.793931 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-137.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 00:33:06.794007 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.793947 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-137.ec2.internal" event="NodeHasSufficientPID" Sep 08 00:33:06.794007 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.793974 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-137.ec2.internal" Sep 08 00:33:06.801724 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.801710 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-137.ec2.internal" Sep 08 00:33:06.801774 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:06.801731 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-137.ec2.internal\": node \"ip-10-0-139-137.ec2.internal\" not found" Sep 08 00:33:06.818409 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:06.818360 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-137.ec2.internal\" not found" Sep 08 00:33:06.884750 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.884717 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-137.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-137.ec2.internal"] Sep 08 00:33:06.884830 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.884799 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 00:33:06.885593 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.885578 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-137.ec2.internal" event="NodeHasSufficientMemory" Sep 08 00:33:06.885685 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.885607 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-137.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 00:33:06.885685 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.885621 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-137.ec2.internal" event="NodeHasSufficientPID" Sep 08 00:33:06.887060 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.887046 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 00:33:06.887194 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.887180 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-137.ec2.internal" Sep 08 00:33:06.887233 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.887208 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 00:33:06.887737 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.887720 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-137.ec2.internal" event="NodeHasSufficientMemory" Sep 08 00:33:06.887825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.887746 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-137.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 00:33:06.887825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.887724 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-137.ec2.internal" event="NodeHasSufficientMemory" Sep 08 00:33:06.887825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.887761 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-137.ec2.internal" event="NodeHasSufficientPID" Sep 08 00:33:06.887825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.887773 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-137.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 00:33:06.887825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.887789 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-137.ec2.internal" event="NodeHasSufficientPID" Sep 08 00:33:06.888959 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.888944 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-137.ec2.internal" Sep 08 00:33:06.889024 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.888971 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 00:33:06.889614 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.889598 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-137.ec2.internal" event="NodeHasSufficientMemory" Sep 08 00:33:06.889681 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.889621 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-137.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 00:33:06.889681 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.889635 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-137.ec2.internal" event="NodeHasSufficientPID" Sep 08 00:33:06.911697 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:06.911681 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-137.ec2.internal\" not found" node="ip-10-0-139-137.ec2.internal" Sep 08 00:33:06.916123 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:06.916110 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-137.ec2.internal\" not found" node="ip-10-0-139-137.ec2.internal" Sep 08 00:33:06.919158 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:06.919141 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-137.ec2.internal\" not found" Sep 08 00:33:06.947192 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.947173 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2db403b7bf9b9056fcc88bd0941837a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-137.ec2.internal\" (UID: \"2db403b7bf9b9056fcc88bd0941837a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-137.ec2.internal" Sep 08 00:33:06.947277 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.947195 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2db403b7bf9b9056fcc88bd0941837a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-137.ec2.internal\" (UID: \"2db403b7bf9b9056fcc88bd0941837a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-137.ec2.internal" Sep 08 00:33:06.947277 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:06.947214 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed147346fb2da2df0b70fefaed4bb0fc-config\") pod \"kube-apiserver-proxy-ip-10-0-139-137.ec2.internal\" (UID: \"ed147346fb2da2df0b70fefaed4bb0fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-137.ec2.internal" Sep 08 00:33:07.019526 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:07.019498 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-137.ec2.internal\" not found" Sep 08 00:33:07.047849 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:07.047808 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2db403b7bf9b9056fcc88bd0941837a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-137.ec2.internal\" (UID: \"2db403b7bf9b9056fcc88bd0941837a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-137.ec2.internal" Sep 08 00:33:07.047968 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:07.047857 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2db403b7bf9b9056fcc88bd0941837a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-137.ec2.internal\" (UID: \"2db403b7bf9b9056fcc88bd0941837a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-137.ec2.internal" Sep 08 00:33:07.047968 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:07.047896 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed147346fb2da2df0b70fefaed4bb0fc-config\") pod \"kube-apiserver-proxy-ip-10-0-139-137.ec2.internal\" (UID: \"ed147346fb2da2df0b70fefaed4bb0fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-137.ec2.internal" Sep 08 00:33:07.047968 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:07.047905 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2db403b7bf9b9056fcc88bd0941837a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-137.ec2.internal\" (UID: \"2db403b7bf9b9056fcc88bd0941837a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-137.ec2.internal" Sep 08 00:33:07.047968 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:07.047927 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ed147346fb2da2df0b70fefaed4bb0fc-config\") pod \"kube-apiserver-proxy-ip-10-0-139-137.ec2.internal\" (UID: \"ed147346fb2da2df0b70fefaed4bb0fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-137.ec2.internal" Sep 08 00:33:07.048123 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:07.047970 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2db403b7bf9b9056fcc88bd0941837a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-137.ec2.internal\" (UID: \"2db403b7bf9b9056fcc88bd0941837a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-137.ec2.internal" Sep 08 00:33:07.120133 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:07.120075 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-137.ec2.internal\" not found" Sep 08 00:33:07.214716 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:07.214681 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-137.ec2.internal" Sep 08 00:33:07.219971 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:07.219956 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-137.ec2.internal" Sep 08 00:33:07.221022 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:07.220986 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-137.ec2.internal\" not found" Sep 08 00:33:07.321445 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:07.321408 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-137.ec2.internal\" not found" Sep 08 00:33:07.421685 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:07.421659 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-137.ec2.internal\" not found" Sep 08 00:33:07.524916 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:07.524877 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-137.ec2.internal\" not found" Sep 08 00:33:07.584239 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:07.584204 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 08 00:33:07.584909 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:07.584354 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 00:33:07.625683 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:07.625648 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-137.ec2.internal\" not found" Sep 08 00:33:07.672352 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:07.672264 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 00:28:07 +0000 UTC" deadline="2028-04-10 23:36:40.709324898 +0000 UTC" Sep 08 00:33:07.672352 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:07.672296 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13943h3m33.037033806s" Sep 08 00:33:07.726691 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:07.726665 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-137.ec2.internal\" not found" Sep 08 00:33:07.732799 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:07.732780 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 00:33:07.745620 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:07.745591 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-137.ec2.internal" Sep 08 00:33:07.761238 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:07.761204 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 00:33:07.763061 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:07.763042 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-137.ec2.internal" Sep 08 00:33:07.772721 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:07.772695 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 00:33:07.824469 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:07.824434 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded147346fb2da2df0b70fefaed4bb0fc.slice/crio-97b602ae34cdf97fe501fa473f4dde53c3dc37540fb50afeb8322dabb07c28ee WatchSource:0}: Error finding container 97b602ae34cdf97fe501fa473f4dde53c3dc37540fb50afeb8322dabb07c28ee: Status 404 returned error can't find the container with id 97b602ae34cdf97fe501fa473f4dde53c3dc37540fb50afeb8322dabb07c28ee Sep 08 00:33:07.824983 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:07.824945 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db403b7bf9b9056fcc88bd0941837a9.slice/crio-16bafc6445644b63e3a028ac6e02cbdbc2ea5c5ccc5c34c8333300effe70b7ea WatchSource:0}: Error finding container 16bafc6445644b63e3a028ac6e02cbdbc2ea5c5ccc5c34c8333300effe70b7ea: Status 404 returned error can't find the container with id 16bafc6445644b63e3a028ac6e02cbdbc2ea5c5ccc5c34c8333300effe70b7ea Sep 08 00:33:07.831986 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:07.831970 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 00:33:08.153377 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.153343 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 00:33:08.215890 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.215860 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 00:33:08.626172 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.626136 2577 apiserver.go:52] "Watching apiserver" Sep 08 00:33:08.633236 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.633209 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 08 00:33:08.633696 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.633663 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-sl7cg","kube-system/kube-apiserver-proxy-ip-10-0-139-137.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz","openshift-dns/node-resolver-zpwqj","openshift-multus/multus-86j4p","openshift-multus/multus-additional-cni-plugins-ph66x","openshift-multus/network-metrics-daemon-5xvxv","openshift-network-diagnostics/network-check-target-dfgn5","kube-system/konnectivity-agent-mxsjc","openshift-cluster-node-tuning-operator/tuned-gzmzw","openshift-image-registry/node-ca-25zw7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-137.ec2.internal","openshift-network-operator/iptables-alerter-fw9mk","openshift-ovn-kubernetes/ovnkube-node-9dm6k"] Sep 08 00:33:08.635394 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.635366 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:08.635519 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:08.635465 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xvxv" podUID="8d449e65-fe9f-4c43-9d0b-4e130446e7a7" Sep 08 00:33:08.637207 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.636777 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fw9mk" Sep 08 00:33:08.638250 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.638230 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.639838 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.639375 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 00:33:08.639838 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.639601 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zpwqj" Sep 08 00:33:08.640321 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.640298 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 08 00:33:08.640813 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.640794 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 08 00:33:08.641019 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.641000 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 08 00:33:08.641244 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.641228 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n9h4v\"" Sep 08 00:33:08.641569 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.641549 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.642953 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.642932 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.644373 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.644355 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mxsjc" Sep 08 00:33:08.645518 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.645227 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 08 00:33:08.645981 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.645905 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:08.646096 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:08.645979 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl7cg" podUID="d30a2b6e-de4c-4204-9427-08784f0330c8" Sep 08 00:33:08.647227 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.646983 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 08 00:33:08.647329 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.647292 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 08 00:33:08.647617 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.647593 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.647988 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.647592 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 08 00:33:08.649699 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.649677 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 08 00:33:08.649805 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.649700 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.649940 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.649924 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 08 00:33:08.650230 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.650215 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 08 00:33:08.650331 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.650312 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 08 00:33:08.650416 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.650388 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 08 00:33:08.650581 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.650568 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 08 00:33:08.650638 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.650603 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 08 00:33:08.650638 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.650615 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 08 00:33:08.650638 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.650571 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-45sc2\"" Sep 08 00:33:08.650788 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.650711 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qctq4\"" Sep 08 00:33:08.650845 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.650827 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 08 00:33:08.650912 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.650896 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jxjq8\"" Sep 08 00:33:08.651006 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.650897 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-25zw7" Sep 08 00:33:08.651097 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.651010 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 08 00:33:08.651176 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.651154 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xzxc5\"" Sep 08 00:33:08.651259 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.650981 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dnbtz\"" Sep 08 00:33:08.652915 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.652897 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:08.652915 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:08.652949 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfgn5" podUID="012969b4-e601-4068-96e9-6f80acb76afb" Sep 08 00:33:08.654247 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.654226 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 08 00:33:08.657407 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656225 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2412f205-2d19-400a-9c85-e0108f6890f5-agent-certs\") pod \"konnectivity-agent-mxsjc\" (UID: \"2412f205-2d19-400a-9c85-e0108f6890f5\") " pod="kube-system/konnectivity-agent-mxsjc" Sep 08 00:33:08.657407 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656270 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d30a2b6e-de4c-4204-9427-08784f0330c8-kubelet-config\") pod \"global-pull-secret-syncer-sl7cg\" (UID: \"d30a2b6e-de4c-4204-9427-08784f0330c8\") " pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:08.657407 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656307 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret\") pod \"global-pull-secret-syncer-sl7cg\" (UID: \"d30a2b6e-de4c-4204-9427-08784f0330c8\") " pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:08.657407 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656341 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d49f2de-3fec-4c6e-806b-7ce6be0667de-host-slash\") pod \"iptables-alerter-fw9mk\" (UID: \"1d49f2de-3fec-4c6e-806b-7ce6be0667de\") " pod="openshift-network-operator/iptables-alerter-fw9mk" Sep 08 00:33:08.657407 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656370 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/50c6132c-861b-4d0e-99b0-3848fd145438-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r52bz\" (UID: \"50c6132c-861b-4d0e-99b0-3848fd145438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.657407 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656402 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/50c6132c-861b-4d0e-99b0-3848fd145438-device-dir\") pod \"aws-ebs-csi-driver-node-r52bz\" (UID: \"50c6132c-861b-4d0e-99b0-3848fd145438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.657407 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656436 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-host-run-k8s-cni-cncf-io\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.657407 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656468 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-host-var-lib-cni-multus\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.657407 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656503 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b9672be-e104-4b0f-a6fd-19861dea1e82-multus-daemon-config\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.657407 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656531 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75c3c9f4-d05c-4eff-8613-08fac22ad5f4-system-cni-dir\") pod \"multus-additional-cni-plugins-ph66x\" (UID: \"75c3c9f4-d05c-4eff-8613-08fac22ad5f4\") " pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.657407 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656564 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75c3c9f4-d05c-4eff-8613-08fac22ad5f4-cnibin\") pod \"multus-additional-cni-plugins-ph66x\" (UID: \"75c3c9f4-d05c-4eff-8613-08fac22ad5f4\") " pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.657407 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656595 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b13bec4-42f2-4fda-95df-9773dfd10505-tmp-dir\") pod \"node-resolver-zpwqj\" (UID: \"2b13bec4-42f2-4fda-95df-9773dfd10505\") " pod="openshift-dns/node-resolver-zpwqj" Sep 08 00:33:08.657407 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656627 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d30a2b6e-de4c-4204-9427-08784f0330c8-dbus\") pod \"global-pull-secret-syncer-sl7cg\" (UID: \"d30a2b6e-de4c-4204-9427-08784f0330c8\") " pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:08.657407 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656661 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq8sw\" (UniqueName: \"kubernetes.io/projected/1d49f2de-3fec-4c6e-806b-7ce6be0667de-kube-api-access-mq8sw\") pod \"iptables-alerter-fw9mk\" (UID: \"1d49f2de-3fec-4c6e-806b-7ce6be0667de\") " pod="openshift-network-operator/iptables-alerter-fw9mk" Sep 08 00:33:08.657407 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656689 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg54k\" (UniqueName: \"kubernetes.io/projected/50c6132c-861b-4d0e-99b0-3848fd145438-kube-api-access-fg54k\") pod \"aws-ebs-csi-driver-node-r52bz\" (UID: \"50c6132c-861b-4d0e-99b0-3848fd145438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.657407 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656723 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-host-var-lib-cni-bin\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.658255 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656758 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j7bn\" (UniqueName: \"kubernetes.io/projected/75c3c9f4-d05c-4eff-8613-08fac22ad5f4-kube-api-access-2j7bn\") pod \"multus-additional-cni-plugins-ph66x\" (UID: \"75c3c9f4-d05c-4eff-8613-08fac22ad5f4\") " pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.658255 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656800 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/50c6132c-861b-4d0e-99b0-3848fd145438-sys-fs\") pod \"aws-ebs-csi-driver-node-r52bz\" (UID: \"50c6132c-861b-4d0e-99b0-3848fd145438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.658255 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656831 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-system-cni-dir\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.658255 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656865 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-host-run-netns\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.658255 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656890 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-hostroot\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.658255 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656923 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75c3c9f4-d05c-4eff-8613-08fac22ad5f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-ph66x\" (UID: \"75c3c9f4-d05c-4eff-8613-08fac22ad5f4\") " pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.658255 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656956 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 08 00:33:08.658255 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.657227 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 08 00:33:08.658255 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.657347 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 08 00:33:08.658255 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.656955 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/75c3c9f4-d05c-4eff-8613-08fac22ad5f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ph66x\" (UID: \"75c3c9f4-d05c-4eff-8613-08fac22ad5f4\") " pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.658255 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.657565 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psvml\" (UniqueName: \"kubernetes.io/projected/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-kube-api-access-psvml\") pod \"network-metrics-daemon-5xvxv\" (UID: \"8d449e65-fe9f-4c43-9d0b-4e130446e7a7\") " pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:08.658255 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.657605 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d49f2de-3fec-4c6e-806b-7ce6be0667de-iptables-alerter-script\") pod \"iptables-alerter-fw9mk\" (UID: \"1d49f2de-3fec-4c6e-806b-7ce6be0667de\") " pod="openshift-network-operator/iptables-alerter-fw9mk" Sep 08 00:33:08.658255 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.657637 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-multus-cni-dir\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.658255 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.657668 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-cnibin\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.658255 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.657719 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-host-run-multus-certs\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.658255 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.657778 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-etc-kubernetes\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.658255 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.657816 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75c3c9f4-d05c-4eff-8613-08fac22ad5f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ph66x\" (UID: \"75c3c9f4-d05c-4eff-8613-08fac22ad5f4\") " pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.658255 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.657851 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2412f205-2d19-400a-9c85-e0108f6890f5-konnectivity-ca\") pod \"konnectivity-agent-mxsjc\" (UID: \"2412f205-2d19-400a-9c85-e0108f6890f5\") " pod="kube-system/konnectivity-agent-mxsjc" Sep 08 00:33:08.659171 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.657915 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/50c6132c-861b-4d0e-99b0-3848fd145438-socket-dir\") pod \"aws-ebs-csi-driver-node-r52bz\" (UID: \"50c6132c-861b-4d0e-99b0-3848fd145438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.659171 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.657953 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-multus-socket-dir-parent\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.659171 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.657983 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx4qh\" (UniqueName: \"kubernetes.io/projected/5b9672be-e104-4b0f-a6fd-19861dea1e82-kube-api-access-sx4qh\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.659171 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.658051 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2b13bec4-42f2-4fda-95df-9773dfd10505-hosts-file\") pod \"node-resolver-zpwqj\" (UID: \"2b13bec4-42f2-4fda-95df-9773dfd10505\") " pod="openshift-dns/node-resolver-zpwqj" Sep 08 00:33:08.659171 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.658105 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs\") pod \"network-metrics-daemon-5xvxv\" (UID: \"8d449e65-fe9f-4c43-9d0b-4e130446e7a7\") " pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:08.659171 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.658143 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/50c6132c-861b-4d0e-99b0-3848fd145438-registration-dir\") pod \"aws-ebs-csi-driver-node-r52bz\" (UID: \"50c6132c-861b-4d0e-99b0-3848fd145438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.659171 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.658169 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/50c6132c-861b-4d0e-99b0-3848fd145438-etc-selinux\") pod \"aws-ebs-csi-driver-node-r52bz\" (UID: \"50c6132c-861b-4d0e-99b0-3848fd145438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.659171 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.658219 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b9672be-e104-4b0f-a6fd-19861dea1e82-cni-binary-copy\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.659171 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.658260 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-host-var-lib-kubelet\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.659171 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.658309 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-multus-conf-dir\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.659171 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.658363 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/75c3c9f4-d05c-4eff-8613-08fac22ad5f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ph66x\" (UID: \"75c3c9f4-d05c-4eff-8613-08fac22ad5f4\") " pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.659171 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.658393 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82cjf\" (UniqueName: \"kubernetes.io/projected/2b13bec4-42f2-4fda-95df-9773dfd10505-kube-api-access-82cjf\") pod \"node-resolver-zpwqj\" (UID: \"2b13bec4-42f2-4fda-95df-9773dfd10505\") " pod="openshift-dns/node-resolver-zpwqj" Sep 08 00:33:08.659913 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.659569 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mln59\"" Sep 08 00:33:08.659913 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.659617 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 08 00:33:08.659913 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.659773 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 08 00:33:08.660064 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.659920 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 08 00:33:08.660150 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.660123 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 08 00:33:08.660203 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.660177 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 08 00:33:08.660261 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.660205 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 08 00:33:08.660261 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.660229 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rf2ln\"" Sep 08 00:33:08.660261 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.660177 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hhc5p\"" Sep 08 00:33:08.660426 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.660233 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 08 00:33:08.668388 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.668368 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 00:33:08.695188 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.695158 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mwlpb" Sep 08 00:33:08.706038 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.706017 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mwlpb" Sep 08 00:33:08.747220 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.747201 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 08 00:33:08.758815 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.758787 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/946f3c40-ec45-41e1-af40-cb6868e839dc-etc-tuned\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.758925 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.758832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2412f205-2d19-400a-9c85-e0108f6890f5-agent-certs\") pod \"konnectivity-agent-mxsjc\" (UID: \"2412f205-2d19-400a-9c85-e0108f6890f5\") " pod="kube-system/konnectivity-agent-mxsjc" Sep 08 00:33:08.758925 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.758855 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d49f2de-3fec-4c6e-806b-7ce6be0667de-host-slash\") pod \"iptables-alerter-fw9mk\" (UID: \"1d49f2de-3fec-4c6e-806b-7ce6be0667de\") " pod="openshift-network-operator/iptables-alerter-fw9mk" Sep 08 00:33:08.758925 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.758880 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/50c6132c-861b-4d0e-99b0-3848fd145438-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r52bz\" (UID: \"50c6132c-861b-4d0e-99b0-3848fd145438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.758925 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.758908 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-host-cni-bin\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.759159 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.758931 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-lib-modules\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.759159 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.758954 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3587d6fa-1292-4006-9971-9937f6de1058-serviceca\") pod \"node-ca-25zw7\" (UID: \"3587d6fa-1292-4006-9971-9937f6de1058\") " pod="openshift-image-registry/node-ca-25zw7" Sep 08 00:33:08.759159 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.758952 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1d49f2de-3fec-4c6e-806b-7ce6be0667de-host-slash\") pod \"iptables-alerter-fw9mk\" (UID: \"1d49f2de-3fec-4c6e-806b-7ce6be0667de\") " pod="openshift-network-operator/iptables-alerter-fw9mk" Sep 08 00:33:08.759159 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.758996 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/50c6132c-861b-4d0e-99b0-3848fd145438-device-dir\") pod \"aws-ebs-csi-driver-node-r52bz\" (UID: \"50c6132c-861b-4d0e-99b0-3848fd145438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.759159 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759064 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b9672be-e104-4b0f-a6fd-19861dea1e82-multus-daemon-config\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.759159 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.758991 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/50c6132c-861b-4d0e-99b0-3848fd145438-kubelet-dir\") pod \"aws-ebs-csi-driver-node-r52bz\" (UID: \"50c6132c-861b-4d0e-99b0-3848fd145438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.759159 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759077 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/50c6132c-861b-4d0e-99b0-3848fd145438-device-dir\") pod \"aws-ebs-csi-driver-node-r52bz\" (UID: \"50c6132c-861b-4d0e-99b0-3848fd145438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.759159 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759090 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75c3c9f4-d05c-4eff-8613-08fac22ad5f4-cnibin\") pod \"multus-additional-cni-plugins-ph66x\" (UID: \"75c3c9f4-d05c-4eff-8613-08fac22ad5f4\") " pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.759159 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759116 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b13bec4-42f2-4fda-95df-9773dfd10505-tmp-dir\") pod \"node-resolver-zpwqj\" (UID: \"2b13bec4-42f2-4fda-95df-9773dfd10505\") " pod="openshift-dns/node-resolver-zpwqj" Sep 08 00:33:08.759159 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759144 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-host-cni-netd\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.759609 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759198 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75c3c9f4-d05c-4eff-8613-08fac22ad5f4-cnibin\") pod \"multus-additional-cni-plugins-ph66x\" (UID: \"75c3c9f4-d05c-4eff-8613-08fac22ad5f4\") " pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.759609 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759240 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq8sw\" (UniqueName: \"kubernetes.io/projected/1d49f2de-3fec-4c6e-806b-7ce6be0667de-kube-api-access-mq8sw\") pod \"iptables-alerter-fw9mk\" (UID: \"1d49f2de-3fec-4c6e-806b-7ce6be0667de\") " pod="openshift-network-operator/iptables-alerter-fw9mk" Sep 08 00:33:08.759609 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759271 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg54k\" (UniqueName: \"kubernetes.io/projected/50c6132c-861b-4d0e-99b0-3848fd145438-kube-api-access-fg54k\") pod \"aws-ebs-csi-driver-node-r52bz\" (UID: \"50c6132c-861b-4d0e-99b0-3848fd145438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.759609 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759320 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j7bn\" (UniqueName: \"kubernetes.io/projected/75c3c9f4-d05c-4eff-8613-08fac22ad5f4-kube-api-access-2j7bn\") pod \"multus-additional-cni-plugins-ph66x\" (UID: \"75c3c9f4-d05c-4eff-8613-08fac22ad5f4\") " pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.759609 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759371 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-host-run-netns\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.759609 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759399 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-env-overrides\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.759609 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759425 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-run\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.759609 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759467 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b13bec4-42f2-4fda-95df-9773dfd10505-tmp-dir\") pod \"node-resolver-zpwqj\" (UID: \"2b13bec4-42f2-4fda-95df-9773dfd10505\") " pod="openshift-dns/node-resolver-zpwqj" Sep 08 00:33:08.759609 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759472 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-system-cni-dir\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.759609 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759520 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-hostroot\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.759609 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759546 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75c3c9f4-d05c-4eff-8613-08fac22ad5f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-ph66x\" (UID: \"75c3c9f4-d05c-4eff-8613-08fac22ad5f4\") " pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.759609 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759569 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/75c3c9f4-d05c-4eff-8613-08fac22ad5f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ph66x\" (UID: \"75c3c9f4-d05c-4eff-8613-08fac22ad5f4\") " pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.759609 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759595 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.760229 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759620 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-host-kubelet\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.760229 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759630 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-system-cni-dir\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.760229 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759645 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-run-openvswitch\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.760229 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759668 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-host\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.760229 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2412f205-2d19-400a-9c85-e0108f6890f5-konnectivity-ca\") pod \"konnectivity-agent-mxsjc\" (UID: \"2412f205-2d19-400a-9c85-e0108f6890f5\") " pod="kube-system/konnectivity-agent-mxsjc" Sep 08 00:33:08.760229 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759712 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5b9672be-e104-4b0f-a6fd-19861dea1e82-multus-daemon-config\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.760229 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759731 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/50c6132c-861b-4d0e-99b0-3848fd145438-socket-dir\") pod \"aws-ebs-csi-driver-node-r52bz\" (UID: \"50c6132c-861b-4d0e-99b0-3848fd145438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.760229 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759762 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-hostroot\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.760229 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759766 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2b13bec4-42f2-4fda-95df-9773dfd10505-hosts-file\") pod \"node-resolver-zpwqj\" (UID: \"2b13bec4-42f2-4fda-95df-9773dfd10505\") " pod="openshift-dns/node-resolver-zpwqj" Sep 08 00:33:08.760229 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759802 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-systemd-units\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.760229 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759843 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hshbt\" (UniqueName: \"kubernetes.io/projected/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-kube-api-access-hshbt\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.760229 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759868 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/946f3c40-ec45-41e1-af40-cb6868e839dc-tmp\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.760229 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759909 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b9672be-e104-4b0f-a6fd-19861dea1e82-cni-binary-copy\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.760229 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759935 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-host-var-lib-kubelet\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.760229 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.759961 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-node-log\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.760229 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760022 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-ovn-node-metrics-cert\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.760229 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760049 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-etc-sysctl-conf\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.760971 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760096 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/75c3c9f4-d05c-4eff-8613-08fac22ad5f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ph66x\" (UID: \"75c3c9f4-d05c-4eff-8613-08fac22ad5f4\") " pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.760971 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760126 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82cjf\" (UniqueName: \"kubernetes.io/projected/2b13bec4-42f2-4fda-95df-9773dfd10505-kube-api-access-82cjf\") pod \"node-resolver-zpwqj\" (UID: \"2b13bec4-42f2-4fda-95df-9773dfd10505\") " pod="openshift-dns/node-resolver-zpwqj" Sep 08 00:33:08.760971 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760157 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-host-slash\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.760971 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760180 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-run-systemd\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.760971 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760178 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 08 00:33:08.760971 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760218 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.760971 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760247 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d30a2b6e-de4c-4204-9427-08784f0330c8-kubelet-config\") pod \"global-pull-secret-syncer-sl7cg\" (UID: \"d30a2b6e-de4c-4204-9427-08784f0330c8\") " pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:08.760971 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760253 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2412f205-2d19-400a-9c85-e0108f6890f5-konnectivity-ca\") pod \"konnectivity-agent-mxsjc\" (UID: \"2412f205-2d19-400a-9c85-e0108f6890f5\") " pod="kube-system/konnectivity-agent-mxsjc" Sep 08 00:33:08.760971 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760274 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret\") pod \"global-pull-secret-syncer-sl7cg\" (UID: \"d30a2b6e-de4c-4204-9427-08784f0330c8\") " pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:08.760971 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760325 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-etc-kubernetes\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.760971 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760340 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2b13bec4-42f2-4fda-95df-9773dfd10505-hosts-file\") pod \"node-resolver-zpwqj\" (UID: \"2b13bec4-42f2-4fda-95df-9773dfd10505\") " pod="openshift-dns/node-resolver-zpwqj" Sep 08 00:33:08.760971 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760354 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-host-run-k8s-cni-cncf-io\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.760971 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760383 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-host-var-lib-cni-multus\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.760971 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760352 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75c3c9f4-d05c-4eff-8613-08fac22ad5f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-ph66x\" (UID: \"75c3c9f4-d05c-4eff-8613-08fac22ad5f4\") " pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.760971 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760352 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/75c3c9f4-d05c-4eff-8613-08fac22ad5f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ph66x\" (UID: \"75c3c9f4-d05c-4eff-8613-08fac22ad5f4\") " pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.760971 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760430 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/50c6132c-861b-4d0e-99b0-3848fd145438-socket-dir\") pod \"aws-ebs-csi-driver-node-r52bz\" (UID: \"50c6132c-861b-4d0e-99b0-3848fd145438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.760971 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760454 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-host-run-k8s-cni-cncf-io\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.761825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760462 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-host-var-lib-cni-multus\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.761825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760464 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-host-var-lib-kubelet\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.761825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760505 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75c3c9f4-d05c-4eff-8613-08fac22ad5f4-system-cni-dir\") pod \"multus-additional-cni-plugins-ph66x\" (UID: \"75c3c9f4-d05c-4eff-8613-08fac22ad5f4\") " pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.761825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760517 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d30a2b6e-de4c-4204-9427-08784f0330c8-kubelet-config\") pod \"global-pull-secret-syncer-sl7cg\" (UID: \"d30a2b6e-de4c-4204-9427-08784f0330c8\") " pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:08.761825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760535 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-log-socket\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.761825 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:08.760539 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 00:33:08.761825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760653 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-etc-modprobe-d\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.761825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760669 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b9672be-e104-4b0f-a6fd-19861dea1e82-cni-binary-copy\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.761825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760679 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-etc-systemd\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.761825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760707 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt75q\" (UniqueName: \"kubernetes.io/projected/946f3c40-ec45-41e1-af40-cb6868e839dc-kube-api-access-zt75q\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.761825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760733 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75c3c9f4-d05c-4eff-8613-08fac22ad5f4-system-cni-dir\") pod \"multus-additional-cni-plugins-ph66x\" (UID: \"75c3c9f4-d05c-4eff-8613-08fac22ad5f4\") " pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.761825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760784 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d30a2b6e-de4c-4204-9427-08784f0330c8-dbus\") pod \"global-pull-secret-syncer-sl7cg\" (UID: \"d30a2b6e-de4c-4204-9427-08784f0330c8\") " pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:08.761825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760814 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-host-var-lib-cni-bin\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.761825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760836 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/75c3c9f4-d05c-4eff-8613-08fac22ad5f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ph66x\" (UID: \"75c3c9f4-d05c-4eff-8613-08fac22ad5f4\") " pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.761825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760842 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-var-lib-kubelet\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.761825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760841 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d30a2b6e-de4c-4204-9427-08784f0330c8-dbus\") pod \"global-pull-secret-syncer-sl7cg\" (UID: \"d30a2b6e-de4c-4204-9427-08784f0330c8\") " pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:08.761825 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:08.760877 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret podName:d30a2b6e-de4c-4204-9427-08784f0330c8 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:09.260856216 +0000 UTC m=+2.956097357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret") pod "global-pull-secret-syncer-sl7cg" (UID: "d30a2b6e-de4c-4204-9427-08784f0330c8") : object "kube-system"/"original-pull-secret" not registered Sep 08 00:33:08.762776 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760884 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-host-var-lib-cni-bin\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.762776 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760913 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj2ft\" (UniqueName: \"kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft\") pod \"network-check-target-dfgn5\" (UID: \"012969b4-e601-4068-96e9-6f80acb76afb\") " pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:08.762776 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760943 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/50c6132c-861b-4d0e-99b0-3848fd145438-sys-fs\") pod \"aws-ebs-csi-driver-node-r52bz\" (UID: \"50c6132c-861b-4d0e-99b0-3848fd145438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.762776 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.760970 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-host-run-netns\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.762776 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761001 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-var-lib-openvswitch\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.762776 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761025 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-run-ovn\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.762776 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761044 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/50c6132c-861b-4d0e-99b0-3848fd145438-sys-fs\") pod \"aws-ebs-csi-driver-node-r52bz\" (UID: \"50c6132c-861b-4d0e-99b0-3848fd145438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.762776 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761057 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-ovnkube-config\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.762776 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761091 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-host-run-netns\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.762776 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761097 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psvml\" (UniqueName: \"kubernetes.io/projected/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-kube-api-access-psvml\") pod \"network-metrics-daemon-5xvxv\" (UID: \"8d449e65-fe9f-4c43-9d0b-4e130446e7a7\") " pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:08.762776 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761140 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d49f2de-3fec-4c6e-806b-7ce6be0667de-iptables-alerter-script\") pod \"iptables-alerter-fw9mk\" (UID: \"1d49f2de-3fec-4c6e-806b-7ce6be0667de\") " pod="openshift-network-operator/iptables-alerter-fw9mk" Sep 08 00:33:08.762776 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761194 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-multus-cni-dir\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.762776 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-cnibin\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.762776 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761259 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-host-run-multus-certs\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.762776 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761300 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-etc-kubernetes\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.762776 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761316 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-host-run-multus-certs\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.762776 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761325 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75c3c9f4-d05c-4eff-8613-08fac22ad5f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ph66x\" (UID: \"75c3c9f4-d05c-4eff-8613-08fac22ad5f4\") " pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.763602 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761361 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-etc-kubernetes\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.763602 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761379 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-cnibin\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.763602 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761394 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3587d6fa-1292-4006-9971-9937f6de1058-host\") pod \"node-ca-25zw7\" (UID: \"3587d6fa-1292-4006-9971-9937f6de1058\") " pod="openshift-image-registry/node-ca-25zw7" Sep 08 00:33:08.763602 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761379 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-multus-cni-dir\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.763602 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761433 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-multus-socket-dir-parent\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.763602 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761475 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx4qh\" (UniqueName: \"kubernetes.io/projected/5b9672be-e104-4b0f-a6fd-19861dea1e82-kube-api-access-sx4qh\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.763602 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761504 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-etc-sysconfig\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.763602 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761515 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75c3c9f4-d05c-4eff-8613-08fac22ad5f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ph66x\" (UID: \"75c3c9f4-d05c-4eff-8613-08fac22ad5f4\") " pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.763602 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761529 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-etc-sysctl-d\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.763602 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761547 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-multus-socket-dir-parent\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.763602 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761557 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58fln\" (UniqueName: \"kubernetes.io/projected/3587d6fa-1292-4006-9971-9937f6de1058-kube-api-access-58fln\") pod \"node-ca-25zw7\" (UID: \"3587d6fa-1292-4006-9971-9937f6de1058\") " pod="openshift-image-registry/node-ca-25zw7" Sep 08 00:33:08.763602 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs\") pod \"network-metrics-daemon-5xvxv\" (UID: \"8d449e65-fe9f-4c43-9d0b-4e130446e7a7\") " pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:08.763602 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761663 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/50c6132c-861b-4d0e-99b0-3848fd145438-registration-dir\") pod \"aws-ebs-csi-driver-node-r52bz\" (UID: \"50c6132c-861b-4d0e-99b0-3848fd145438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.763602 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761690 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/50c6132c-861b-4d0e-99b0-3848fd145438-etc-selinux\") pod \"aws-ebs-csi-driver-node-r52bz\" (UID: \"50c6132c-861b-4d0e-99b0-3848fd145438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.763602 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761742 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/50c6132c-861b-4d0e-99b0-3848fd145438-etc-selinux\") pod \"aws-ebs-csi-driver-node-r52bz\" (UID: \"50c6132c-861b-4d0e-99b0-3848fd145438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.763602 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761744 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-multus-conf-dir\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.763602 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761748 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/50c6132c-861b-4d0e-99b0-3848fd145438-registration-dir\") pod \"aws-ebs-csi-driver-node-r52bz\" (UID: \"50c6132c-861b-4d0e-99b0-3848fd145438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.765990 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761752 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1d49f2de-3fec-4c6e-806b-7ce6be0667de-iptables-alerter-script\") pod \"iptables-alerter-fw9mk\" (UID: \"1d49f2de-3fec-4c6e-806b-7ce6be0667de\") " pod="openshift-network-operator/iptables-alerter-fw9mk" Sep 08 00:33:08.765990 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:08.761812 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 00:33:08.765990 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761852 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-etc-openvswitch\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.765990 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761855 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b9672be-e104-4b0f-a6fd-19861dea1e82-multus-conf-dir\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.765990 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:08.761883 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs podName:8d449e65-fe9f-4c43-9d0b-4e130446e7a7 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:09.261869169 +0000 UTC m=+2.957110307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs") pod "network-metrics-daemon-5xvxv" (UID: "8d449e65-fe9f-4c43-9d0b-4e130446e7a7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 00:33:08.765990 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761902 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-ovnkube-script-lib\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.765990 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.761923 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-sys\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.765990 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.764517 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2412f205-2d19-400a-9c85-e0108f6890f5-agent-certs\") pod \"konnectivity-agent-mxsjc\" (UID: \"2412f205-2d19-400a-9c85-e0108f6890f5\") " pod="kube-system/konnectivity-agent-mxsjc" Sep 08 00:33:08.771341 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.771317 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j7bn\" (UniqueName: \"kubernetes.io/projected/75c3c9f4-d05c-4eff-8613-08fac22ad5f4-kube-api-access-2j7bn\") pod \"multus-additional-cni-plugins-ph66x\" (UID: \"75c3c9f4-d05c-4eff-8613-08fac22ad5f4\") " pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.775750 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.775723 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg54k\" (UniqueName: \"kubernetes.io/projected/50c6132c-861b-4d0e-99b0-3848fd145438-kube-api-access-fg54k\") pod \"aws-ebs-csi-driver-node-r52bz\" (UID: \"50c6132c-861b-4d0e-99b0-3848fd145438\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.777148 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.777124 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psvml\" (UniqueName: \"kubernetes.io/projected/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-kube-api-access-psvml\") pod \"network-metrics-daemon-5xvxv\" (UID: \"8d449e65-fe9f-4c43-9d0b-4e130446e7a7\") " pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:08.777235 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.777205 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx4qh\" (UniqueName: \"kubernetes.io/projected/5b9672be-e104-4b0f-a6fd-19861dea1e82-kube-api-access-sx4qh\") pod \"multus-86j4p\" (UID: \"5b9672be-e104-4b0f-a6fd-19861dea1e82\") " pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.777834 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.777810 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq8sw\" (UniqueName: \"kubernetes.io/projected/1d49f2de-3fec-4c6e-806b-7ce6be0667de-kube-api-access-mq8sw\") pod \"iptables-alerter-fw9mk\" (UID: \"1d49f2de-3fec-4c6e-806b-7ce6be0667de\") " pod="openshift-network-operator/iptables-alerter-fw9mk" Sep 08 00:33:08.779034 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.779001 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82cjf\" (UniqueName: \"kubernetes.io/projected/2b13bec4-42f2-4fda-95df-9773dfd10505-kube-api-access-82cjf\") pod \"node-resolver-zpwqj\" (UID: \"2b13bec4-42f2-4fda-95df-9773dfd10505\") " pod="openshift-dns/node-resolver-zpwqj" Sep 08 00:33:08.789682 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.789590 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-137.ec2.internal" event={"ID":"2db403b7bf9b9056fcc88bd0941837a9","Type":"ContainerStarted","Data":"16bafc6445644b63e3a028ac6e02cbdbc2ea5c5ccc5c34c8333300effe70b7ea"} Sep 08 00:33:08.791049 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.791024 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-137.ec2.internal" event={"ID":"ed147346fb2da2df0b70fefaed4bb0fc","Type":"ContainerStarted","Data":"97b602ae34cdf97fe501fa473f4dde53c3dc37540fb50afeb8322dabb07c28ee"} Sep 08 00:33:08.862237 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862205 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-host-cni-bin\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.862237 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862236 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-lib-modules\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.862486 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862269 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3587d6fa-1292-4006-9971-9937f6de1058-serviceca\") pod \"node-ca-25zw7\" (UID: \"3587d6fa-1292-4006-9971-9937f6de1058\") " pod="openshift-image-registry/node-ca-25zw7" Sep 08 00:33:08.862486 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862297 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-host-cni-netd\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.862486 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862325 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-host-run-netns\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.862486 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862331 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-host-cni-bin\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.862486 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862346 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-env-overrides\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.862486 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862367 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-run\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.862486 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862394 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.862486 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862419 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-host-kubelet\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.862486 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862421 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-lib-modules\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.862486 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862432 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-host-run-netns\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.862486 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862443 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-run-openvswitch\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.862486 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862477 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-host-cni-netd\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.862486 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862491 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-run-openvswitch\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.862930 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862512 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-run\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.862930 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862527 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.862930 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862539 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-host-kubelet\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.862930 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862566 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-host\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.862930 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862600 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-systemd-units\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.862930 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hshbt\" (UniqueName: \"kubernetes.io/projected/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-kube-api-access-hshbt\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.862930 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/946f3c40-ec45-41e1-af40-cb6868e839dc-tmp\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.862930 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-node-log\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.862930 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862677 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-host\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.862930 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862687 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-systemd-units\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.862930 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862701 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-ovn-node-metrics-cert\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.862930 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862729 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-etc-sysctl-conf\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.862930 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862740 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3587d6fa-1292-4006-9971-9937f6de1058-serviceca\") pod \"node-ca-25zw7\" (UID: \"3587d6fa-1292-4006-9971-9937f6de1058\") " pod="openshift-image-registry/node-ca-25zw7" Sep 08 00:33:08.862930 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862757 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-host-slash\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.862930 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862782 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-run-systemd\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.862930 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862808 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.862930 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862849 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-etc-kubernetes\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.862930 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862876 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-log-socket\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.863878 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862898 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-etc-modprobe-d\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.863878 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862939 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-etc-systemd\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.863878 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862924 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-env-overrides\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.863878 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862967 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt75q\" (UniqueName: \"kubernetes.io/projected/946f3c40-ec45-41e1-af40-cb6868e839dc-kube-api-access-zt75q\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.863878 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862977 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.863878 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.862995 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-var-lib-kubelet\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.863878 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863002 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-log-socket\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.863878 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863022 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-node-log\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.863878 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863028 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj2ft\" (UniqueName: \"kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft\") pod \"network-check-target-dfgn5\" (UID: \"012969b4-e601-4068-96e9-6f80acb76afb\") " pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:08.863878 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863088 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-etc-kubernetes\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.863878 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-var-lib-openvswitch\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.863878 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863101 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-etc-systemd\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.863878 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863133 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-run-ovn\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.863878 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863046 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-host-slash\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.863878 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863134 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-var-lib-openvswitch\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.863878 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863159 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-ovnkube-config\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.863878 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863166 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-run-systemd\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.864769 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863135 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-etc-sysctl-conf\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.864769 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863193 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3587d6fa-1292-4006-9971-9937f6de1058-host\") pod \"node-ca-25zw7\" (UID: \"3587d6fa-1292-4006-9971-9937f6de1058\") " pod="openshift-image-registry/node-ca-25zw7" Sep 08 00:33:08.864769 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863199 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-etc-modprobe-d\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.864769 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863219 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-etc-sysconfig\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.864769 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863226 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-run-ovn\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.864769 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863239 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3587d6fa-1292-4006-9971-9937f6de1058-host\") pod \"node-ca-25zw7\" (UID: \"3587d6fa-1292-4006-9971-9937f6de1058\") " pod="openshift-image-registry/node-ca-25zw7" Sep 08 00:33:08.864769 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863253 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-etc-sysctl-d\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.864769 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863279 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58fln\" (UniqueName: \"kubernetes.io/projected/3587d6fa-1292-4006-9971-9937f6de1058-kube-api-access-58fln\") pod \"node-ca-25zw7\" (UID: \"3587d6fa-1292-4006-9971-9937f6de1058\") " pod="openshift-image-registry/node-ca-25zw7" Sep 08 00:33:08.864769 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863285 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-etc-sysconfig\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.864769 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863322 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-etc-openvswitch\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.864769 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863317 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-var-lib-kubelet\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.864769 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863348 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-ovnkube-script-lib\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.864769 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863370 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-sys\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.864769 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863405 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/946f3c40-ec45-41e1-af40-cb6868e839dc-etc-tuned\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.864769 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863413 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-etc-openvswitch\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.864769 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863380 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-etc-sysctl-d\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.864769 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863437 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/946f3c40-ec45-41e1-af40-cb6868e839dc-sys\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.864769 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.863655 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-ovnkube-config\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.865685 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.864087 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-ovnkube-script-lib\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.865685 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.865616 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/946f3c40-ec45-41e1-af40-cb6868e839dc-tmp\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.865685 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.865672 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-ovn-node-metrics-cert\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.866349 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.866326 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/946f3c40-ec45-41e1-af40-cb6868e839dc-etc-tuned\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.869934 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:08.869910 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 00:33:08.869934 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:08.869938 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 00:33:08.870096 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:08.869952 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lj2ft for pod openshift-network-diagnostics/network-check-target-dfgn5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 00:33:08.870096 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:08.870041 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft podName:012969b4-e601-4068-96e9-6f80acb76afb nodeName:}" failed. No retries permitted until 2026-09-08 00:33:09.370007228 +0000 UTC m=+3.065248373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lj2ft" (UniqueName: "kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft") pod "network-check-target-dfgn5" (UID: "012969b4-e601-4068-96e9-6f80acb76afb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 00:33:08.873188 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.873168 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hshbt\" (UniqueName: \"kubernetes.io/projected/ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9-kube-api-access-hshbt\") pod \"ovnkube-node-9dm6k\" (UID: \"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:08.873470 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.873449 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58fln\" (UniqueName: \"kubernetes.io/projected/3587d6fa-1292-4006-9971-9937f6de1058-kube-api-access-58fln\") pod \"node-ca-25zw7\" (UID: \"3587d6fa-1292-4006-9971-9937f6de1058\") " pod="openshift-image-registry/node-ca-25zw7" Sep 08 00:33:08.873470 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.873465 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt75q\" (UniqueName: \"kubernetes.io/projected/946f3c40-ec45-41e1-af40-cb6868e839dc-kube-api-access-zt75q\") pod \"tuned-gzmzw\" (UID: \"946f3c40-ec45-41e1-af40-cb6868e839dc\") " pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:08.952834 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.952751 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fw9mk" Sep 08 00:33:08.965257 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.965228 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" Sep 08 00:33:08.977219 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.977188 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zpwqj" Sep 08 00:33:08.982425 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.982401 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-86j4p" Sep 08 00:33:08.989443 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.989421 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ph66x" Sep 08 00:33:08.997321 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:08.997300 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mxsjc" Sep 08 00:33:09.006110 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:09.006088 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:09.013990 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:09.013970 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" Sep 08 00:33:09.020781 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:09.020762 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-25zw7" Sep 08 00:33:09.267434 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:09.267320 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs\") pod \"network-metrics-daemon-5xvxv\" (UID: \"8d449e65-fe9f-4c43-9d0b-4e130446e7a7\") " pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:09.267434 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:09.267397 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret\") pod \"global-pull-secret-syncer-sl7cg\" (UID: \"d30a2b6e-de4c-4204-9427-08784f0330c8\") " pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:09.267661 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:09.267536 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 00:33:09.267661 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:09.267555 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 00:33:09.267661 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:09.267603 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret podName:d30a2b6e-de4c-4204-9427-08784f0330c8 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:10.267584717 +0000 UTC m=+3.962825862 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret") pod "global-pull-secret-syncer-sl7cg" (UID: "d30a2b6e-de4c-4204-9427-08784f0330c8") : object "kube-system"/"original-pull-secret" not registered Sep 08 00:33:09.267661 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:09.267623 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs podName:8d449e65-fe9f-4c43-9d0b-4e130446e7a7 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:10.267613454 +0000 UTC m=+3.962854592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs") pod "network-metrics-daemon-5xvxv" (UID: "8d449e65-fe9f-4c43-9d0b-4e130446e7a7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 00:33:09.469343 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:09.469308 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj2ft\" (UniqueName: \"kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft\") pod \"network-check-target-dfgn5\" (UID: \"012969b4-e601-4068-96e9-6f80acb76afb\") " pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:09.469536 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:09.469474 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 00:33:09.469536 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:09.469496 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 00:33:09.469536 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:09.469506 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lj2ft for pod openshift-network-diagnostics/network-check-target-dfgn5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 00:33:09.469643 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:09.469554 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft podName:012969b4-e601-4068-96e9-6f80acb76afb nodeName:}" failed. No retries permitted until 2026-09-08 00:33:10.469540864 +0000 UTC m=+4.164782008 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lj2ft" (UniqueName: "kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft") pod "network-check-target-dfgn5" (UID: "012969b4-e601-4068-96e9-6f80acb76afb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 00:33:09.708866 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:09.708824 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 00:28:09 +0000 UTC" deadline="2028-05-15 17:46:18.478296988 +0000 UTC" Sep 08 00:33:09.708866 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:09.708858 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14777h13m8.769442473s" Sep 08 00:33:09.784745 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:09.784715 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:09.784928 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:09.784847 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl7cg" podUID="d30a2b6e-de4c-4204-9427-08784f0330c8" Sep 08 00:33:09.905441 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:09.905406 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3587d6fa_1292_4006_9971_9937f6de1058.slice/crio-1e8f1f8f79e945a8700373d41ce1ff08170a9d8f192caab2d47baca067d10164 WatchSource:0}: Error finding container 1e8f1f8f79e945a8700373d41ce1ff08170a9d8f192caab2d47baca067d10164: Status 404 returned error can't find the container with id 1e8f1f8f79e945a8700373d41ce1ff08170a9d8f192caab2d47baca067d10164 Sep 08 00:33:09.907336 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:09.907230 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946f3c40_ec45_41e1_af40_cb6868e839dc.slice/crio-2e4c2c940282a7381f930fb0bf624794eb8920b430debf9bad194d22dc489f74 WatchSource:0}: Error finding container 2e4c2c940282a7381f930fb0bf624794eb8920b430debf9bad194d22dc489f74: Status 404 returned error can't find the container with id 2e4c2c940282a7381f930fb0bf624794eb8920b430debf9bad194d22dc489f74 Sep 08 00:33:09.908331 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:09.908302 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d49f2de_3fec_4c6e_806b_7ce6be0667de.slice/crio-20b64b98693b6b931a3a50c14808c837803341f7ffdb0401c6d366e0b077ae14 WatchSource:0}: Error finding container 20b64b98693b6b931a3a50c14808c837803341f7ffdb0401c6d366e0b077ae14: Status 404 returned error can't find the container with id 20b64b98693b6b931a3a50c14808c837803341f7ffdb0401c6d366e0b077ae14 Sep 08 00:33:09.913378 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:09.913357 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba181c63_1f88_43f7_a9b4_ab8a5ad3b6f9.slice/crio-3f4b9ab7e2df94af31c981e5a4401f81c96fb7beeba21edb3e3d22333710b43e WatchSource:0}: Error finding container 3f4b9ab7e2df94af31c981e5a4401f81c96fb7beeba21edb3e3d22333710b43e: Status 404 returned error can't find the container with id 3f4b9ab7e2df94af31c981e5a4401f81c96fb7beeba21edb3e3d22333710b43e Sep 08 00:33:09.913969 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:09.913892 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c6132c_861b_4d0e_99b0_3848fd145438.slice/crio-d40d2ba8881fcaec5e9477e47e1b227e091f14e088133ded023d603819cb6c50 WatchSource:0}: Error finding container d40d2ba8881fcaec5e9477e47e1b227e091f14e088133ded023d603819cb6c50: Status 404 returned error can't find the container with id d40d2ba8881fcaec5e9477e47e1b227e091f14e088133ded023d603819cb6c50 Sep 08 00:33:09.917050 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:09.917028 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b9672be_e104_4b0f_a6fd_19861dea1e82.slice/crio-d8916622dd783671651e9ec77e74e37c4381d864f3d1cfe94765e63d356bfc17 WatchSource:0}: Error finding container d8916622dd783671651e9ec77e74e37c4381d864f3d1cfe94765e63d356bfc17: Status 404 returned error can't find the container with id d8916622dd783671651e9ec77e74e37c4381d864f3d1cfe94765e63d356bfc17 Sep 08 00:33:09.918591 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:09.917679 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c3c9f4_d05c_4eff_8613_08fac22ad5f4.slice/crio-59331e54d97e3f583ab87a6b21819a963c8ca46c3ab2f62ef7f7d03fe1bced0e WatchSource:0}: Error finding container 59331e54d97e3f583ab87a6b21819a963c8ca46c3ab2f62ef7f7d03fe1bced0e: Status 404 returned error can't find the container with id 59331e54d97e3f583ab87a6b21819a963c8ca46c3ab2f62ef7f7d03fe1bced0e Sep 08 00:33:10.276180 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:10.275975 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs\") pod \"network-metrics-daemon-5xvxv\" (UID: \"8d449e65-fe9f-4c43-9d0b-4e130446e7a7\") " pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:10.276322 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:10.276244 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret\") pod \"global-pull-secret-syncer-sl7cg\" (UID: \"d30a2b6e-de4c-4204-9427-08784f0330c8\") " pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:10.276322 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:10.276255 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 00:33:10.276404 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:10.276359 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 00:33:10.276404 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:10.276368 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs podName:8d449e65-fe9f-4c43-9d0b-4e130446e7a7 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:12.276341151 +0000 UTC m=+5.971582308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs") pod "network-metrics-daemon-5xvxv" (UID: "8d449e65-fe9f-4c43-9d0b-4e130446e7a7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 00:33:10.276404 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:10.276401 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret podName:d30a2b6e-de4c-4204-9427-08784f0330c8 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:12.276387694 +0000 UTC m=+5.971628832 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret") pod "global-pull-secret-syncer-sl7cg" (UID: "d30a2b6e-de4c-4204-9427-08784f0330c8") : object "kube-system"/"original-pull-secret" not registered Sep 08 00:33:10.477613 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:10.477567 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj2ft\" (UniqueName: \"kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft\") pod \"network-check-target-dfgn5\" (UID: \"012969b4-e601-4068-96e9-6f80acb76afb\") " pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:10.477825 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:10.477797 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 00:33:10.477825 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:10.477820 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 00:33:10.477952 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:10.477835 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lj2ft for pod openshift-network-diagnostics/network-check-target-dfgn5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 00:33:10.477952 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:10.477897 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft podName:012969b4-e601-4068-96e9-6f80acb76afb nodeName:}" failed. No retries permitted until 2026-09-08 00:33:12.47787784 +0000 UTC m=+6.173118993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lj2ft" (UniqueName: "kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft") pod "network-check-target-dfgn5" (UID: "012969b4-e601-4068-96e9-6f80acb76afb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 00:33:10.709814 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:10.709740 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 00:28:09 +0000 UTC" deadline="2028-02-14 05:27:41.451470459 +0000 UTC" Sep 08 00:33:10.709814 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:10.709804 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12580h54m30.741670382s" Sep 08 00:33:10.786756 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:10.786192 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:10.786756 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:10.786321 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfgn5" podUID="012969b4-e601-4068-96e9-6f80acb76afb" Sep 08 00:33:10.786756 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:10.786563 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:10.786756 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:10.786709 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xvxv" podUID="8d449e65-fe9f-4c43-9d0b-4e130446e7a7" Sep 08 00:33:10.842114 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:10.842051 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-137.ec2.internal" event={"ID":"ed147346fb2da2df0b70fefaed4bb0fc","Type":"ContainerStarted","Data":"1162c12a4f3248e362e5baaad51eb31e80f75fe24cdb3d4d5303c53715e6e874"} Sep 08 00:33:10.850791 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:10.850759 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" event={"ID":"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9","Type":"ContainerStarted","Data":"3f4b9ab7e2df94af31c981e5a4401f81c96fb7beeba21edb3e3d22333710b43e"} Sep 08 00:33:10.867955 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:10.867873 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ph66x" event={"ID":"75c3c9f4-d05c-4eff-8613-08fac22ad5f4","Type":"ContainerStarted","Data":"59331e54d97e3f583ab87a6b21819a963c8ca46c3ab2f62ef7f7d03fe1bced0e"} Sep 08 00:33:10.873892 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:10.873847 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" event={"ID":"50c6132c-861b-4d0e-99b0-3848fd145438","Type":"ContainerStarted","Data":"d40d2ba8881fcaec5e9477e47e1b227e091f14e088133ded023d603819cb6c50"} Sep 08 00:33:10.886522 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:10.886449 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fw9mk" event={"ID":"1d49f2de-3fec-4c6e-806b-7ce6be0667de","Type":"ContainerStarted","Data":"20b64b98693b6b931a3a50c14808c837803341f7ffdb0401c6d366e0b077ae14"} Sep 08 00:33:10.909409 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:10.909372 2577 generic.go:356] "Generic (PLEG): container finished" podID="2db403b7bf9b9056fcc88bd0941837a9" containerID="cbd51db22f8c6c12aed88f3ad3643414a75f5d09d30dbed9d92a28bf0c09c7e9" exitCode=0 Sep 08 00:33:10.909699 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:10.909628 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-137.ec2.internal" event={"ID":"2db403b7bf9b9056fcc88bd0941837a9","Type":"ContainerDied","Data":"cbd51db22f8c6c12aed88f3ad3643414a75f5d09d30dbed9d92a28bf0c09c7e9"} Sep 08 00:33:10.915784 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:10.915700 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mxsjc" event={"ID":"2412f205-2d19-400a-9c85-e0108f6890f5","Type":"ContainerStarted","Data":"bb8f98cf769e31892bddd7de8893465ebd621752bcea7c6dbf3c8b4d16d47f2d"} Sep 08 00:33:10.927004 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:10.926936 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-137.ec2.internal" podStartSLOduration=2.926916787 podStartE2EDuration="2.926916787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:10.864276331 +0000 UTC m=+4.559517495" watchObservedRunningTime="2026-09-08 00:33:10.926916787 +0000 UTC m=+4.622157953" Sep 08 00:33:10.928635 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:10.928587 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-86j4p" event={"ID":"5b9672be-e104-4b0f-a6fd-19861dea1e82","Type":"ContainerStarted","Data":"d8916622dd783671651e9ec77e74e37c4381d864f3d1cfe94765e63d356bfc17"} Sep 08 00:33:10.941455 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:10.941428 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zpwqj" event={"ID":"2b13bec4-42f2-4fda-95df-9773dfd10505","Type":"ContainerStarted","Data":"75dbd66800c8ad20577dfb778ad7f177ba165251d07fcc811f0462d0bf95de05"} Sep 08 00:33:10.948594 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:10.948565 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" event={"ID":"946f3c40-ec45-41e1-af40-cb6868e839dc","Type":"ContainerStarted","Data":"2e4c2c940282a7381f930fb0bf624794eb8920b430debf9bad194d22dc489f74"} Sep 08 00:33:10.954647 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:10.954595 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-25zw7" event={"ID":"3587d6fa-1292-4006-9971-9937f6de1058","Type":"ContainerStarted","Data":"1e8f1f8f79e945a8700373d41ce1ff08170a9d8f192caab2d47baca067d10164"} Sep 08 00:33:11.784584 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:11.784545 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:11.785225 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:11.784680 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl7cg" podUID="d30a2b6e-de4c-4204-9427-08784f0330c8" Sep 08 00:33:11.974279 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:11.974169 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-137.ec2.internal" event={"ID":"2db403b7bf9b9056fcc88bd0941837a9","Type":"ContainerStarted","Data":"2d8eed957d0bbd26412a0b44be3521c81e82802f78198453e126a05c9a4ea516"} Sep 08 00:33:12.293676 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:12.293636 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs\") pod \"network-metrics-daemon-5xvxv\" (UID: \"8d449e65-fe9f-4c43-9d0b-4e130446e7a7\") " pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:12.293871 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:12.293727 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret\") pod \"global-pull-secret-syncer-sl7cg\" (UID: \"d30a2b6e-de4c-4204-9427-08784f0330c8\") " pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:12.293871 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:12.293815 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 00:33:12.293871 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:12.293855 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 00:33:12.294053 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:12.293907 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs podName:8d449e65-fe9f-4c43-9d0b-4e130446e7a7 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:16.293883403 +0000 UTC m=+9.989124545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs") pod "network-metrics-daemon-5xvxv" (UID: "8d449e65-fe9f-4c43-9d0b-4e130446e7a7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 00:33:12.294053 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:12.293928 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret podName:d30a2b6e-de4c-4204-9427-08784f0330c8 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:16.293918066 +0000 UTC m=+9.989159211 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret") pod "global-pull-secret-syncer-sl7cg" (UID: "d30a2b6e-de4c-4204-9427-08784f0330c8") : object "kube-system"/"original-pull-secret" not registered Sep 08 00:33:12.496395 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:12.495672 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj2ft\" (UniqueName: \"kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft\") pod \"network-check-target-dfgn5\" (UID: \"012969b4-e601-4068-96e9-6f80acb76afb\") " pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:12.496395 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:12.495870 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 00:33:12.496395 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:12.495895 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 00:33:12.496395 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:12.495908 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lj2ft for pod openshift-network-diagnostics/network-check-target-dfgn5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 00:33:12.496395 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:12.495968 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft podName:012969b4-e601-4068-96e9-6f80acb76afb nodeName:}" failed. No retries permitted until 2026-09-08 00:33:16.495949963 +0000 UTC m=+10.191191120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lj2ft" (UniqueName: "kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft") pod "network-check-target-dfgn5" (UID: "012969b4-e601-4068-96e9-6f80acb76afb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 00:33:12.788275 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:12.787490 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:12.788275 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:12.787619 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfgn5" podUID="012969b4-e601-4068-96e9-6f80acb76afb" Sep 08 00:33:12.788275 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:12.788015 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:12.788275 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:12.788146 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xvxv" podUID="8d449e65-fe9f-4c43-9d0b-4e130446e7a7" Sep 08 00:33:13.785392 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:13.785359 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:13.785578 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:13.785495 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl7cg" podUID="d30a2b6e-de4c-4204-9427-08784f0330c8" Sep 08 00:33:14.784703 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:14.784665 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:14.785197 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:14.784817 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfgn5" podUID="012969b4-e601-4068-96e9-6f80acb76afb" Sep 08 00:33:14.785281 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:14.785261 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:14.785397 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:14.785371 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xvxv" podUID="8d449e65-fe9f-4c43-9d0b-4e130446e7a7" Sep 08 00:33:15.784610 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:15.784575 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:15.784783 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:15.784734 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl7cg" podUID="d30a2b6e-de4c-4204-9427-08784f0330c8" Sep 08 00:33:16.327582 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:16.327531 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret\") pod \"global-pull-secret-syncer-sl7cg\" (UID: \"d30a2b6e-de4c-4204-9427-08784f0330c8\") " pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:16.327794 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:16.327691 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs\") pod \"network-metrics-daemon-5xvxv\" (UID: \"8d449e65-fe9f-4c43-9d0b-4e130446e7a7\") " pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:16.327794 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:16.327722 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 00:33:16.327917 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:16.327806 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret podName:d30a2b6e-de4c-4204-9427-08784f0330c8 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:24.327781512 +0000 UTC m=+18.023022653 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret") pod "global-pull-secret-syncer-sl7cg" (UID: "d30a2b6e-de4c-4204-9427-08784f0330c8") : object "kube-system"/"original-pull-secret" not registered Sep 08 00:33:16.327917 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:16.327837 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 00:33:16.327917 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:16.327893 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs podName:8d449e65-fe9f-4c43-9d0b-4e130446e7a7 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:24.327876832 +0000 UTC m=+18.023117988 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs") pod "network-metrics-daemon-5xvxv" (UID: "8d449e65-fe9f-4c43-9d0b-4e130446e7a7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 00:33:16.529643 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:16.529600 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj2ft\" (UniqueName: \"kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft\") pod \"network-check-target-dfgn5\" (UID: \"012969b4-e601-4068-96e9-6f80acb76afb\") " pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:16.529829 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:16.529801 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 00:33:16.529890 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:16.529839 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 00:33:16.529890 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:16.529854 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lj2ft for pod openshift-network-diagnostics/network-check-target-dfgn5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 00:33:16.529988 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:16.529943 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft podName:012969b4-e601-4068-96e9-6f80acb76afb nodeName:}" failed. No retries permitted until 2026-09-08 00:33:24.529924518 +0000 UTC m=+18.225165677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lj2ft" (UniqueName: "kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft") pod "network-check-target-dfgn5" (UID: "012969b4-e601-4068-96e9-6f80acb76afb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 00:33:16.785668 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:16.785632 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:16.786190 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:16.785737 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:16.786190 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:16.785761 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfgn5" podUID="012969b4-e601-4068-96e9-6f80acb76afb" Sep 08 00:33:16.786190 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:16.785873 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xvxv" podUID="8d449e65-fe9f-4c43-9d0b-4e130446e7a7" Sep 08 00:33:17.785605 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:17.785563 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:17.785845 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:17.785720 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl7cg" podUID="d30a2b6e-de4c-4204-9427-08784f0330c8" Sep 08 00:33:18.785423 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:18.785058 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:18.785423 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:18.785185 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfgn5" podUID="012969b4-e601-4068-96e9-6f80acb76afb" Sep 08 00:33:18.785423 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:18.785237 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:18.785423 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:18.785348 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xvxv" podUID="8d449e65-fe9f-4c43-9d0b-4e130446e7a7" Sep 08 00:33:19.784701 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:19.784669 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:19.785147 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:19.784812 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl7cg" podUID="d30a2b6e-de4c-4204-9427-08784f0330c8" Sep 08 00:33:20.784578 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:20.784550 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:20.784714 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:20.784675 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfgn5" podUID="012969b4-e601-4068-96e9-6f80acb76afb" Sep 08 00:33:20.785045 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:20.784735 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:20.785045 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:20.784866 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xvxv" podUID="8d449e65-fe9f-4c43-9d0b-4e130446e7a7" Sep 08 00:33:21.784541 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:21.784505 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:21.784727 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:21.784617 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl7cg" podUID="d30a2b6e-de4c-4204-9427-08784f0330c8" Sep 08 00:33:22.784702 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:22.784667 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:22.784702 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:22.784698 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:22.785214 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:22.784787 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfgn5" podUID="012969b4-e601-4068-96e9-6f80acb76afb" Sep 08 00:33:22.785214 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:22.784943 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xvxv" podUID="8d449e65-fe9f-4c43-9d0b-4e130446e7a7" Sep 08 00:33:23.785222 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:23.785191 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:23.785715 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:23.785308 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl7cg" podUID="d30a2b6e-de4c-4204-9427-08784f0330c8" Sep 08 00:33:24.395736 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:24.395705 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs\") pod \"network-metrics-daemon-5xvxv\" (UID: \"8d449e65-fe9f-4c43-9d0b-4e130446e7a7\") " pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:24.395922 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:24.395768 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret\") pod \"global-pull-secret-syncer-sl7cg\" (UID: \"d30a2b6e-de4c-4204-9427-08784f0330c8\") " pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:24.395922 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:24.395838 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 00:33:24.395922 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:24.395861 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 00:33:24.395922 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:24.395899 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs podName:8d449e65-fe9f-4c43-9d0b-4e130446e7a7 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:40.395883241 +0000 UTC m=+34.091124383 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs") pod "network-metrics-daemon-5xvxv" (UID: "8d449e65-fe9f-4c43-9d0b-4e130446e7a7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 00:33:24.395922 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:24.395913 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret podName:d30a2b6e-de4c-4204-9427-08784f0330c8 nodeName:}" failed. No retries permitted until 2026-09-08 00:33:40.395906663 +0000 UTC m=+34.091147801 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret") pod "global-pull-secret-syncer-sl7cg" (UID: "d30a2b6e-de4c-4204-9427-08784f0330c8") : object "kube-system"/"original-pull-secret" not registered Sep 08 00:33:24.597723 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:24.597692 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj2ft\" (UniqueName: \"kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft\") pod \"network-check-target-dfgn5\" (UID: \"012969b4-e601-4068-96e9-6f80acb76afb\") " pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:24.597877 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:24.597807 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 00:33:24.597877 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:24.597822 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 00:33:24.597877 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:24.597831 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lj2ft for pod openshift-network-diagnostics/network-check-target-dfgn5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 00:33:24.597984 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:24.597880 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft podName:012969b4-e601-4068-96e9-6f80acb76afb nodeName:}" failed. No retries permitted until 2026-09-08 00:33:40.597864863 +0000 UTC m=+34.293106000 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lj2ft" (UniqueName: "kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft") pod "network-check-target-dfgn5" (UID: "012969b4-e601-4068-96e9-6f80acb76afb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 00:33:24.785230 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:24.785206 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:24.785585 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:24.785212 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:24.785585 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:24.785310 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfgn5" podUID="012969b4-e601-4068-96e9-6f80acb76afb" Sep 08 00:33:24.785585 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:24.785374 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xvxv" podUID="8d449e65-fe9f-4c43-9d0b-4e130446e7a7" Sep 08 00:33:25.784845 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:25.784809 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:25.785039 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:25.784934 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl7cg" podUID="d30a2b6e-de4c-4204-9427-08784f0330c8" Sep 08 00:33:26.786157 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:26.786112 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:26.786636 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:26.786277 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xvxv" podUID="8d449e65-fe9f-4c43-9d0b-4e130446e7a7" Sep 08 00:33:26.786636 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:26.786299 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:26.786636 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:26.786405 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfgn5" podUID="012969b4-e601-4068-96e9-6f80acb76afb" Sep 08 00:33:27.785180 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:27.785144 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:27.785352 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:27.785262 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl7cg" podUID="d30a2b6e-de4c-4204-9427-08784f0330c8" Sep 08 00:33:28.787822 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:28.787539 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:28.788206 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:28.787608 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:28.788206 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:28.787892 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfgn5" podUID="012969b4-e601-4068-96e9-6f80acb76afb" Sep 08 00:33:28.788206 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:28.787971 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xvxv" podUID="8d449e65-fe9f-4c43-9d0b-4e130446e7a7" Sep 08 00:33:29.009727 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:29.009638 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mxsjc" event={"ID":"2412f205-2d19-400a-9c85-e0108f6890f5","Type":"ContainerStarted","Data":"1ea1c6501d3f0daa2f57ba14671cc898d38a8ed92d08705540d191411308344c"} Sep 08 00:33:29.011584 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:29.011558 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-86j4p" event={"ID":"5b9672be-e104-4b0f-a6fd-19861dea1e82","Type":"ContainerStarted","Data":"78b81aea14768060d2cecd60cabfb1454646ebd456b1f4d3a96133105a1d6b39"} Sep 08 00:33:29.013225 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:29.013146 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zpwqj" event={"ID":"2b13bec4-42f2-4fda-95df-9773dfd10505","Type":"ContainerStarted","Data":"4a309537191be94fd5950e6ef8ad569aadbbe15644f046c706e42e9db95e3dde"} Sep 08 00:33:29.014903 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:29.014877 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" event={"ID":"946f3c40-ec45-41e1-af40-cb6868e839dc","Type":"ContainerStarted","Data":"1b0f6871b541f09d3dadcbbe41e1f4af84a6001ada1605f485885520dc7c901e"} Sep 08 00:33:29.016416 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:29.016393 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-25zw7" event={"ID":"3587d6fa-1292-4006-9971-9937f6de1058","Type":"ContainerStarted","Data":"e47f100fd7d6778c170ec44a7491d1d86be5c1cc8fa6e6e5140155bf3d3ef24b"} Sep 08 00:33:29.020032 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:29.019987 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" event={"ID":"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9","Type":"ContainerStarted","Data":"d621c14205ddfb156862c7ebe7773d2756ca76bdf1ebe13c83e7a7e97d52a05e"} Sep 08 00:33:29.020115 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:29.020066 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" event={"ID":"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9","Type":"ContainerStarted","Data":"82944fe91682e3c611493a2128051e0eb3645d4930e0afb262f1545fc60c0891"} Sep 08 00:33:29.020115 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:29.020098 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" event={"ID":"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9","Type":"ContainerStarted","Data":"ae8e7d475fb448a54d4ed0c687033d1c1435bdbbdd54ad5d7caa1579bf348cfe"} Sep 08 00:33:29.020187 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:29.020126 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" event={"ID":"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9","Type":"ContainerStarted","Data":"d66f832034306d66fbbbeab1222f711a60cd250770f857fe9ddda10c8be38a00"} Sep 08 00:33:29.020187 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:29.020153 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" event={"ID":"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9","Type":"ContainerStarted","Data":"d3c1e5af6b43333a2b0ce83b92a3068a7db02a9147946f3bbc07f8e95d072025"} Sep 08 00:33:29.020187 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:29.020180 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" event={"ID":"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9","Type":"ContainerStarted","Data":"e5cea13ed9bbef862a7277d9913bf7d63d6322730b200346e5606a9751835644"} Sep 08 00:33:29.022076 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:29.022053 2577 generic.go:356] "Generic (PLEG): container finished" podID="75c3c9f4-d05c-4eff-8613-08fac22ad5f4" containerID="ab6d9cbb6e33c1e77a87574178370832729f18a550497e5680a9eca1f4c17338" exitCode=0 Sep 08 00:33:29.022181 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:29.022079 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ph66x" event={"ID":"75c3c9f4-d05c-4eff-8613-08fac22ad5f4","Type":"ContainerDied","Data":"ab6d9cbb6e33c1e77a87574178370832729f18a550497e5680a9eca1f4c17338"} Sep 08 00:33:29.024072 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:29.024050 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" event={"ID":"50c6132c-861b-4d0e-99b0-3848fd145438","Type":"ContainerStarted","Data":"8df2e5c81747550df467e3b867f3fa02b987d7cb0822fd2c4f493911f7e1e238"} Sep 08 00:33:29.026473 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:29.026429 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mxsjc" podStartSLOduration=3.801695737 podStartE2EDuration="22.026411954s" totalImagesPullingTime="18.224716217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:29.025810995 +0000 UTC m=+22.721052155" watchObservedRunningTime="2026-09-08 00:33:29.026411954 +0000 UTC m=+22.721653113" Sep 08 00:33:29.026659 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:29.026625 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-137.ec2.internal" podStartSLOduration=21.026614689 podStartE2EDuration="21.026614689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:12.003435244 +0000 UTC m=+5.698676409" watchObservedRunningTime="2026-09-08 00:33:29.026614689 +0000 UTC m=+22.721855849" Sep 08 00:33:29.044629 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:29.044592 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gzmzw" podStartSLOduration=3.7448717179999997 podStartE2EDuration="22.044581733s" totalImagesPullingTime="18.299710015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:29.044438891 +0000 UTC m=+22.739680051" watchObservedRunningTime="2026-09-08 00:33:29.044581733 +0000 UTC m=+22.739822891" Sep 08 00:33:29.061647 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:29.061605 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zpwqj" podStartSLOduration=3.761990975 podStartE2EDuration="22.061589071s" totalImagesPullingTime="18.299598096s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:29.061116688 +0000 UTC m=+22.756357849" watchObservedRunningTime="2026-09-08 00:33:29.061589071 +0000 UTC m=+22.756830231" Sep 08 00:33:29.075870 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:29.075828 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-25zw7" podStartSLOduration=3.775547905 podStartE2EDuration="22.075812517s" totalImagesPullingTime="18.300264612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:29.075712043 +0000 UTC m=+22.770953203" watchObservedRunningTime="2026-09-08 00:33:29.075812517 +0000 UTC m=+22.771053677" Sep 08 00:33:29.094284 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:29.094242 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-86j4p" podStartSLOduration=3.619252355 podStartE2EDuration="22.094228736s" totalImagesPullingTime="18.474976381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:29.09393609 +0000 UTC m=+22.789177252" watchObservedRunningTime="2026-09-08 00:33:29.094228736 +0000 UTC m=+22.789469894" Sep 08 00:33:29.784931 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:29.784894 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:29.785125 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:29.785011 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl7cg" podUID="d30a2b6e-de4c-4204-9427-08784f0330c8" Sep 08 00:33:29.800608 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:29.800582 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 08 00:33:30.028500 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:30.028467 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" event={"ID":"50c6132c-861b-4d0e-99b0-3848fd145438","Type":"ContainerStarted","Data":"b687b57e7ea5345138fb7ca48ddeb9d21b2c0bfc1bac449dccdcf4e13c137d77"} Sep 08 00:33:30.029844 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:30.029818 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fw9mk" event={"ID":"1d49f2de-3fec-4c6e-806b-7ce6be0667de","Type":"ContainerStarted","Data":"5a6377f1a18eb148b55f8159c62bf833e444caec9e0cf91d7371a826cf7ef086"} Sep 08 00:33:30.044368 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:30.044329 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fw9mk" podStartSLOduration=4.80618239 podStartE2EDuration="23.044312771s" totalImagesPullingTime="18.238130381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:30.044217611 +0000 UTC m=+23.739458770" watchObservedRunningTime="2026-09-08 00:33:30.044312771 +0000 UTC m=+23.739553933" Sep 08 00:33:30.706430 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:30.706339 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T00:33:29.800603539Z","UUID":"2925a386-c2d0-4f11-9c7e-7f6d8acbbfc5","Handler":null,"Name":"","Endpoint":""} Sep 08 00:33:30.707900 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:30.707880 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 08 00:33:30.708017 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:30.707908 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 08 00:33:30.785403 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:30.785367 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:30.785569 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:30.785491 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfgn5" podUID="012969b4-e601-4068-96e9-6f80acb76afb" Sep 08 00:33:30.785569 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:30.785559 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:30.785740 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:30.785685 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xvxv" podUID="8d449e65-fe9f-4c43-9d0b-4e130446e7a7" Sep 08 00:33:31.311013 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:31.310964 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mxsjc" Sep 08 00:33:31.311664 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:31.311645 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mxsjc" Sep 08 00:33:31.785442 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:31.785416 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:31.785613 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:31.785535 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl7cg" podUID="d30a2b6e-de4c-4204-9427-08784f0330c8" Sep 08 00:33:32.039080 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:32.038833 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" event={"ID":"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9","Type":"ContainerStarted","Data":"7576bddb959365cf07a2465cbe4e19e6b891356760f36b66837f99dd174c00eb"} Sep 08 00:33:32.041169 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:32.041135 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" event={"ID":"50c6132c-861b-4d0e-99b0-3848fd145438","Type":"ContainerStarted","Data":"08ee07312ed12613b6f61ea9709ac16e828d8d5ba48e446fdb1547816fb82ece"} Sep 08 00:33:32.058351 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:32.058303 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-r52bz" podStartSLOduration=4.562769039 podStartE2EDuration="25.058285609s" totalImagesPullingTime="20.49551657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:33:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:32.058077739 +0000 UTC m=+25.753318898" watchObservedRunningTime="2026-09-08 00:33:32.058285609 +0000 UTC m=+25.753526769" Sep 08 00:33:32.785181 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:32.785150 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:32.785649 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:32.785150 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:32.785649 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:32.785274 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfgn5" podUID="012969b4-e601-4068-96e9-6f80acb76afb" Sep 08 00:33:32.785649 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:32.785359 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xvxv" podUID="8d449e65-fe9f-4c43-9d0b-4e130446e7a7" Sep 08 00:33:33.785286 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:33.785249 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:33.785679 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:33.785382 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl7cg" podUID="d30a2b6e-de4c-4204-9427-08784f0330c8" Sep 08 00:33:34.785072 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:34.785047 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:34.785167 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:34.785047 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:34.785232 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:34.785176 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xvxv" podUID="8d449e65-fe9f-4c43-9d0b-4e130446e7a7" Sep 08 00:33:34.785285 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:34.785271 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfgn5" podUID="012969b4-e601-4068-96e9-6f80acb76afb" Sep 08 00:33:35.051953 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:35.051751 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" event={"ID":"ba181c63-1f88-43f7-a9b4-ab8a5ad3b6f9","Type":"ContainerStarted","Data":"a458d93c50085c4c5573aa8c9534d0102944865221ca032fe5ad0a55f4ed2acb"} Sep 08 00:33:35.052474 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:35.052027 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:35.052474 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:35.052066 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:35.053813 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:35.053790 2577 generic.go:356] "Generic (PLEG): container finished" podID="75c3c9f4-d05c-4eff-8613-08fac22ad5f4" containerID="9b23e03a524e44872c9fbe2fe49debd15d9bc901105a49f9d57a6b1ccbdcb135" exitCode=0 Sep 08 00:33:35.053927 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:35.053824 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ph66x" event={"ID":"75c3c9f4-d05c-4eff-8613-08fac22ad5f4","Type":"ContainerDied","Data":"9b23e03a524e44872c9fbe2fe49debd15d9bc901105a49f9d57a6b1ccbdcb135"} Sep 08 00:33:35.067352 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:35.067332 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:35.081668 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:35.081625 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" podStartSLOduration=9.575734884 podStartE2EDuration="28.081612567s" totalImagesPullingTime="18.505877683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:35.080413171 +0000 UTC m=+28.775654328" watchObservedRunningTime="2026-09-08 00:33:35.081612567 +0000 UTC m=+28.776853724" Sep 08 00:33:35.784969 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:35.784938 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:35.785217 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:35.785096 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl7cg" podUID="d30a2b6e-de4c-4204-9427-08784f0330c8" Sep 08 00:33:36.056468 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:36.056388 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:36.071601 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:36.071572 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:33:36.770562 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:36.770532 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dfgn5"] Sep 08 00:33:36.770735 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:36.770657 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:36.770803 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:36.770748 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfgn5" podUID="012969b4-e601-4068-96e9-6f80acb76afb" Sep 08 00:33:36.771070 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:36.771049 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5xvxv"] Sep 08 00:33:36.771149 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:36.771135 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:36.771220 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:36.771205 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xvxv" podUID="8d449e65-fe9f-4c43-9d0b-4e130446e7a7" Sep 08 00:33:36.774475 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:36.774453 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sl7cg"] Sep 08 00:33:36.774564 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:36.774531 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:36.774631 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:36.774613 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl7cg" podUID="d30a2b6e-de4c-4204-9427-08784f0330c8" Sep 08 00:33:37.059810 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:37.059780 2577 generic.go:356] "Generic (PLEG): container finished" podID="75c3c9f4-d05c-4eff-8613-08fac22ad5f4" containerID="719e2aab60dd4d4d59fd79ffcb258d6a6467edc9b3319b330a5ae9e01417b4ce" exitCode=0 Sep 08 00:33:37.060455 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:37.059865 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ph66x" event={"ID":"75c3c9f4-d05c-4eff-8613-08fac22ad5f4","Type":"ContainerDied","Data":"719e2aab60dd4d4d59fd79ffcb258d6a6467edc9b3319b330a5ae9e01417b4ce"} Sep 08 00:33:37.576223 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:37.576194 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mxsjc" Sep 08 00:33:37.576371 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:37.576324 2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 08 00:33:37.576884 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:37.576857 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mxsjc" Sep 08 00:33:38.064844 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:38.064598 2577 generic.go:356] "Generic (PLEG): container finished" podID="75c3c9f4-d05c-4eff-8613-08fac22ad5f4" containerID="a0252da6f1962b4efccd3a7c1d7bfd16ff2e809d4463e3bb828013d7795c8a6e" exitCode=0 Sep 08 00:33:38.065247 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:38.064671 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ph66x" event={"ID":"75c3c9f4-d05c-4eff-8613-08fac22ad5f4","Type":"ContainerDied","Data":"a0252da6f1962b4efccd3a7c1d7bfd16ff2e809d4463e3bb828013d7795c8a6e"} Sep 08 00:33:38.785131 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:38.785099 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:38.785286 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:38.785099 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:38.785286 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:38.785223 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xvxv" podUID="8d449e65-fe9f-4c43-9d0b-4e130446e7a7" Sep 08 00:33:38.785286 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:38.785099 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:38.785403 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:38.785298 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl7cg" podUID="d30a2b6e-de4c-4204-9427-08784f0330c8" Sep 08 00:33:38.785403 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:38.785350 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfgn5" podUID="012969b4-e601-4068-96e9-6f80acb76afb" Sep 08 00:33:40.414584 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:40.414522 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret\") pod \"global-pull-secret-syncer-sl7cg\" (UID: \"d30a2b6e-de4c-4204-9427-08784f0330c8\") " pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:40.414584 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:40.414583 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs\") pod \"network-metrics-daemon-5xvxv\" (UID: \"8d449e65-fe9f-4c43-9d0b-4e130446e7a7\") " pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:40.415089 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:40.414681 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 00:33:40.415089 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:40.414749 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs podName:8d449e65-fe9f-4c43-9d0b-4e130446e7a7 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:12.414730054 +0000 UTC m=+66.109971209 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs") pod "network-metrics-daemon-5xvxv" (UID: "8d449e65-fe9f-4c43-9d0b-4e130446e7a7") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 00:33:40.415089 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:40.414682 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 00:33:40.415089 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:40.414791 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret podName:d30a2b6e-de4c-4204-9427-08784f0330c8 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:12.414784537 +0000 UTC m=+66.110025674 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret") pod "global-pull-secret-syncer-sl7cg" (UID: "d30a2b6e-de4c-4204-9427-08784f0330c8") : object "kube-system"/"original-pull-secret" not registered Sep 08 00:33:40.616205 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:40.616168 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj2ft\" (UniqueName: \"kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft\") pod \"network-check-target-dfgn5\" (UID: \"012969b4-e601-4068-96e9-6f80acb76afb\") " pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:40.616378 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:40.616359 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 00:33:40.616452 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:40.616388 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 00:33:40.616452 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:40.616402 2577 projected.go:196] Error preparing data for projected volume kube-api-access-lj2ft for pod openshift-network-diagnostics/network-check-target-dfgn5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 00:33:40.616555 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:40.616472 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft podName:012969b4-e601-4068-96e9-6f80acb76afb nodeName:}" failed. No retries permitted until 2026-09-08 00:34:12.616453317 +0000 UTC m=+66.311694469 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lj2ft" (UniqueName: "kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft") pod "network-check-target-dfgn5" (UID: "012969b4-e601-4068-96e9-6f80acb76afb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 00:33:40.785364 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:40.785052 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:40.785364 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:40.785089 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:40.785364 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:40.785054 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:40.785364 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:40.785199 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5xvxv" podUID="8d449e65-fe9f-4c43-9d0b-4e130446e7a7" Sep 08 00:33:40.785364 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:40.785305 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sl7cg" podUID="d30a2b6e-de4c-4204-9427-08784f0330c8" Sep 08 00:33:40.785967 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:40.785375 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dfgn5" podUID="012969b4-e601-4068-96e9-6f80acb76afb" Sep 08 00:33:41.032658 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.032624 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-137.ec2.internal" event="NodeReady" Sep 08 00:33:41.032828 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.032770 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 08 00:33:41.073340 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.073260 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kfd8x"] Sep 08 00:33:41.119801 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.119770 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xvxpk"] Sep 08 00:33:41.119969 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.119919 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kfd8x" Sep 08 00:33:41.122108 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.122088 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 08 00:33:41.122231 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.122147 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bs5xs\"" Sep 08 00:33:41.122289 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.122270 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 08 00:33:41.131139 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.131120 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kfd8x"] Sep 08 00:33:41.131218 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.131151 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xvxpk"] Sep 08 00:33:41.131260 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.131218 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xvxpk" Sep 08 00:33:41.133706 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.133689 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 08 00:33:41.133830 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.133809 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 08 00:33:41.133912 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.133809 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2f7c4\"" Sep 08 00:33:41.134051 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.134021 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 08 00:33:41.220865 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.220837 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rj6p\" (UniqueName: \"kubernetes.io/projected/08d6201a-e98a-45e5-a580-8d3ab6ca3ef5-kube-api-access-8rj6p\") pod \"ingress-canary-xvxpk\" (UID: \"08d6201a-e98a-45e5-a580-8d3ab6ca3ef5\") " pod="openshift-ingress-canary/ingress-canary-xvxpk" Sep 08 00:33:41.221075 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.220908 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/840e6533-8a38-426a-bb9a-abc4cad33caa-config-volume\") pod \"dns-default-kfd8x\" (UID: \"840e6533-8a38-426a-bb9a-abc4cad33caa\") " pod="openshift-dns/dns-default-kfd8x" Sep 08 00:33:41.221075 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.220955 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08d6201a-e98a-45e5-a580-8d3ab6ca3ef5-cert\") pod \"ingress-canary-xvxpk\" (UID: \"08d6201a-e98a-45e5-a580-8d3ab6ca3ef5\") " pod="openshift-ingress-canary/ingress-canary-xvxpk" Sep 08 00:33:41.221075 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.221020 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/840e6533-8a38-426a-bb9a-abc4cad33caa-metrics-tls\") pod \"dns-default-kfd8x\" (UID: \"840e6533-8a38-426a-bb9a-abc4cad33caa\") " pod="openshift-dns/dns-default-kfd8x" Sep 08 00:33:41.221075 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.221045 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/840e6533-8a38-426a-bb9a-abc4cad33caa-tmp-dir\") pod \"dns-default-kfd8x\" (UID: \"840e6533-8a38-426a-bb9a-abc4cad33caa\") " pod="openshift-dns/dns-default-kfd8x" Sep 08 00:33:41.221075 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.221064 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ckjv\" (UniqueName: \"kubernetes.io/projected/840e6533-8a38-426a-bb9a-abc4cad33caa-kube-api-access-4ckjv\") pod \"dns-default-kfd8x\" (UID: \"840e6533-8a38-426a-bb9a-abc4cad33caa\") " pod="openshift-dns/dns-default-kfd8x" Sep 08 00:33:41.321785 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.321748 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ckjv\" (UniqueName: \"kubernetes.io/projected/840e6533-8a38-426a-bb9a-abc4cad33caa-kube-api-access-4ckjv\") pod \"dns-default-kfd8x\" (UID: \"840e6533-8a38-426a-bb9a-abc4cad33caa\") " pod="openshift-dns/dns-default-kfd8x" Sep 08 00:33:41.321785 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.321787 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rj6p\" (UniqueName: \"kubernetes.io/projected/08d6201a-e98a-45e5-a580-8d3ab6ca3ef5-kube-api-access-8rj6p\") pod \"ingress-canary-xvxpk\" (UID: \"08d6201a-e98a-45e5-a580-8d3ab6ca3ef5\") " pod="openshift-ingress-canary/ingress-canary-xvxpk" Sep 08 00:33:41.322146 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.321830 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/840e6533-8a38-426a-bb9a-abc4cad33caa-config-volume\") pod \"dns-default-kfd8x\" (UID: \"840e6533-8a38-426a-bb9a-abc4cad33caa\") " pod="openshift-dns/dns-default-kfd8x" Sep 08 00:33:41.322146 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.321897 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08d6201a-e98a-45e5-a580-8d3ab6ca3ef5-cert\") pod \"ingress-canary-xvxpk\" (UID: \"08d6201a-e98a-45e5-a580-8d3ab6ca3ef5\") " pod="openshift-ingress-canary/ingress-canary-xvxpk" Sep 08 00:33:41.322146 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.321940 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/840e6533-8a38-426a-bb9a-abc4cad33caa-metrics-tls\") pod \"dns-default-kfd8x\" (UID: \"840e6533-8a38-426a-bb9a-abc4cad33caa\") " pod="openshift-dns/dns-default-kfd8x" Sep 08 00:33:41.322146 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.321961 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/840e6533-8a38-426a-bb9a-abc4cad33caa-tmp-dir\") pod \"dns-default-kfd8x\" (UID: \"840e6533-8a38-426a-bb9a-abc4cad33caa\") " pod="openshift-dns/dns-default-kfd8x" Sep 08 00:33:41.322146 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:41.322058 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 00:33:41.322405 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:41.322117 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/840e6533-8a38-426a-bb9a-abc4cad33caa-metrics-tls podName:840e6533-8a38-426a-bb9a-abc4cad33caa nodeName:}" failed. No retries permitted until 2026-09-08 00:33:41.822099304 +0000 UTC m=+35.517340449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/840e6533-8a38-426a-bb9a-abc4cad33caa-metrics-tls") pod "dns-default-kfd8x" (UID: "840e6533-8a38-426a-bb9a-abc4cad33caa") : secret "dns-default-metrics-tls" not found Sep 08 00:33:41.322405 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.322278 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/840e6533-8a38-426a-bb9a-abc4cad33caa-tmp-dir\") pod \"dns-default-kfd8x\" (UID: \"840e6533-8a38-426a-bb9a-abc4cad33caa\") " pod="openshift-dns/dns-default-kfd8x" Sep 08 00:33:41.322541 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.322481 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/840e6533-8a38-426a-bb9a-abc4cad33caa-config-volume\") pod \"dns-default-kfd8x\" (UID: \"840e6533-8a38-426a-bb9a-abc4cad33caa\") " pod="openshift-dns/dns-default-kfd8x" Sep 08 00:33:41.326687 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.326634 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08d6201a-e98a-45e5-a580-8d3ab6ca3ef5-cert\") pod \"ingress-canary-xvxpk\" (UID: \"08d6201a-e98a-45e5-a580-8d3ab6ca3ef5\") " pod="openshift-ingress-canary/ingress-canary-xvxpk" Sep 08 00:33:41.330665 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.330636 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ckjv\" (UniqueName: \"kubernetes.io/projected/840e6533-8a38-426a-bb9a-abc4cad33caa-kube-api-access-4ckjv\") pod \"dns-default-kfd8x\" (UID: \"840e6533-8a38-426a-bb9a-abc4cad33caa\") " pod="openshift-dns/dns-default-kfd8x" Sep 08 00:33:41.330946 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.330921 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rj6p\" (UniqueName: \"kubernetes.io/projected/08d6201a-e98a-45e5-a580-8d3ab6ca3ef5-kube-api-access-8rj6p\") pod \"ingress-canary-xvxpk\" (UID: \"08d6201a-e98a-45e5-a580-8d3ab6ca3ef5\") " pod="openshift-ingress-canary/ingress-canary-xvxpk" Sep 08 00:33:41.441127 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.441086 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xvxpk" Sep 08 00:33:41.825960 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:41.825920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/840e6533-8a38-426a-bb9a-abc4cad33caa-metrics-tls\") pod \"dns-default-kfd8x\" (UID: \"840e6533-8a38-426a-bb9a-abc4cad33caa\") " pod="openshift-dns/dns-default-kfd8x" Sep 08 00:33:41.826143 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:41.826085 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 00:33:41.826209 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:41.826150 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/840e6533-8a38-426a-bb9a-abc4cad33caa-metrics-tls podName:840e6533-8a38-426a-bb9a-abc4cad33caa nodeName:}" failed. No retries permitted until 2026-09-08 00:33:42.826129698 +0000 UTC m=+36.521370836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/840e6533-8a38-426a-bb9a-abc4cad33caa-metrics-tls") pod "dns-default-kfd8x" (UID: "840e6533-8a38-426a-bb9a-abc4cad33caa") : secret "dns-default-metrics-tls" not found Sep 08 00:33:42.784774 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:42.784735 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:33:42.785247 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:42.784787 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:33:42.785247 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:42.784735 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:33:42.789081 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:42.788639 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 00:33:42.789081 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:42.788661 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 00:33:42.789081 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:42.788678 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wspvj\"" Sep 08 00:33:42.789081 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:42.788820 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5xvx4\"" Sep 08 00:33:42.789081 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:42.788829 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 00:33:42.789081 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:42.788918 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 00:33:42.833720 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:42.833695 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/840e6533-8a38-426a-bb9a-abc4cad33caa-metrics-tls\") pod \"dns-default-kfd8x\" (UID: \"840e6533-8a38-426a-bb9a-abc4cad33caa\") " pod="openshift-dns/dns-default-kfd8x" Sep 08 00:33:42.833855 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:42.833838 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 00:33:42.833906 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:42.833897 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/840e6533-8a38-426a-bb9a-abc4cad33caa-metrics-tls podName:840e6533-8a38-426a-bb9a-abc4cad33caa nodeName:}" failed. No retries permitted until 2026-09-08 00:33:44.833882292 +0000 UTC m=+38.529123428 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/840e6533-8a38-426a-bb9a-abc4cad33caa-metrics-tls") pod "dns-default-kfd8x" (UID: "840e6533-8a38-426a-bb9a-abc4cad33caa") : secret "dns-default-metrics-tls" not found Sep 08 00:33:43.076356 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:43.076171 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xvxpk"] Sep 08 00:33:43.079296 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:43.079269 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d6201a_e98a_45e5_a580_8d3ab6ca3ef5.slice/crio-7cb13ee854dc9ff0498d752cdb319f2990b34c665bc14f66203c08121a8c5ee8 WatchSource:0}: Error finding container 7cb13ee854dc9ff0498d752cdb319f2990b34c665bc14f66203c08121a8c5ee8: Status 404 returned error can't find the container with id 7cb13ee854dc9ff0498d752cdb319f2990b34c665bc14f66203c08121a8c5ee8 Sep 08 00:33:44.084169 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:44.084131 2577 generic.go:356] "Generic (PLEG): container finished" podID="75c3c9f4-d05c-4eff-8613-08fac22ad5f4" containerID="2b87122c98447ab77429edd5817ac07b6a903255ac9eacf8b432f80059a622f9" exitCode=0 Sep 08 00:33:44.084853 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:44.084214 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ph66x" event={"ID":"75c3c9f4-d05c-4eff-8613-08fac22ad5f4","Type":"ContainerDied","Data":"2b87122c98447ab77429edd5817ac07b6a903255ac9eacf8b432f80059a622f9"} Sep 08 00:33:44.085552 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:44.085528 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xvxpk" event={"ID":"08d6201a-e98a-45e5-a580-8d3ab6ca3ef5","Type":"ContainerStarted","Data":"7cb13ee854dc9ff0498d752cdb319f2990b34c665bc14f66203c08121a8c5ee8"} Sep 08 00:33:44.848789 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:44.848755 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/840e6533-8a38-426a-bb9a-abc4cad33caa-metrics-tls\") pod \"dns-default-kfd8x\" (UID: \"840e6533-8a38-426a-bb9a-abc4cad33caa\") " pod="openshift-dns/dns-default-kfd8x" Sep 08 00:33:44.848944 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:44.848878 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 00:33:44.848944 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:44.848932 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/840e6533-8a38-426a-bb9a-abc4cad33caa-metrics-tls podName:840e6533-8a38-426a-bb9a-abc4cad33caa nodeName:}" failed. No retries permitted until 2026-09-08 00:33:48.84891671 +0000 UTC m=+42.544157848 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/840e6533-8a38-426a-bb9a-abc4cad33caa-metrics-tls") pod "dns-default-kfd8x" (UID: "840e6533-8a38-426a-bb9a-abc4cad33caa") : secret "dns-default-metrics-tls" not found Sep 08 00:33:45.091311 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:45.091279 2577 generic.go:356] "Generic (PLEG): container finished" podID="75c3c9f4-d05c-4eff-8613-08fac22ad5f4" containerID="f47499b4b233e329c9a3965b4acf1a186e3f2e573e53c9720fd3fe3a6d6cc8c5" exitCode=0 Sep 08 00:33:45.091757 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:45.091328 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ph66x" event={"ID":"75c3c9f4-d05c-4eff-8613-08fac22ad5f4","Type":"ContainerDied","Data":"f47499b4b233e329c9a3965b4acf1a186e3f2e573e53c9720fd3fe3a6d6cc8c5"} Sep 08 00:33:46.095771 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:46.095742 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xvxpk" event={"ID":"08d6201a-e98a-45e5-a580-8d3ab6ca3ef5","Type":"ContainerStarted","Data":"5e6b8ceac6df7c3a89d8a62ec3a3c89746389da651c781dc9dc26706440f699d"} Sep 08 00:33:46.099659 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:46.099631 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ph66x" event={"ID":"75c3c9f4-d05c-4eff-8613-08fac22ad5f4","Type":"ContainerStarted","Data":"6ba04463b7090816d25b4cd4e1e84b2934ab16398ba716842601d12070c5c686"} Sep 08 00:33:46.115435 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:46.115394 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xvxpk" podStartSLOduration=2.371338587 podStartE2EDuration="5.115379209s" totalImagesPullingTime="2.744040622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:46.114712614 +0000 UTC m=+39.809953772" watchObservedRunningTime="2026-09-08 00:33:46.115379209 +0000 UTC m=+39.810620364" Sep 08 00:33:46.137710 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:46.137626 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ph66x" podStartSLOduration=8.599849167 podStartE2EDuration="39.137610288s" totalImagesPullingTime="30.537761121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:33:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:33:46.136277674 +0000 UTC m=+39.831518832" watchObservedRunningTime="2026-09-08 00:33:46.137610288 +0000 UTC m=+39.832851447" Sep 08 00:33:48.878206 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:48.878151 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/840e6533-8a38-426a-bb9a-abc4cad33caa-metrics-tls\") pod \"dns-default-kfd8x\" (UID: \"840e6533-8a38-426a-bb9a-abc4cad33caa\") " pod="openshift-dns/dns-default-kfd8x" Sep 08 00:33:48.878601 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:48.878303 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 00:33:48.878601 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:48.878362 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/840e6533-8a38-426a-bb9a-abc4cad33caa-metrics-tls podName:840e6533-8a38-426a-bb9a-abc4cad33caa nodeName:}" failed. No retries permitted until 2026-09-08 00:33:56.878347531 +0000 UTC m=+50.573588672 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/840e6533-8a38-426a-bb9a-abc4cad33caa-metrics-tls") pod "dns-default-kfd8x" (UID: "840e6533-8a38-426a-bb9a-abc4cad33caa") : secret "dns-default-metrics-tls" not found Sep 08 00:33:56.935744 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:56.935695 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/840e6533-8a38-426a-bb9a-abc4cad33caa-metrics-tls\") pod \"dns-default-kfd8x\" (UID: \"840e6533-8a38-426a-bb9a-abc4cad33caa\") " pod="openshift-dns/dns-default-kfd8x" Sep 08 00:33:56.936264 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:56.935823 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 00:33:56.936264 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:56.935907 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/840e6533-8a38-426a-bb9a-abc4cad33caa-metrics-tls podName:840e6533-8a38-426a-bb9a-abc4cad33caa nodeName:}" failed. No retries permitted until 2026-09-08 00:34:12.935884422 +0000 UTC m=+66.631125565 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/840e6533-8a38-426a-bb9a-abc4cad33caa-metrics-tls") pod "dns-default-kfd8x" (UID: "840e6533-8a38-426a-bb9a-abc4cad33caa") : secret "dns-default-metrics-tls" not found Sep 08 00:33:58.959861 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:58.959820 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-fgxt2"] Sep 08 00:33:58.973944 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:58.973903 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-9nb9x"] Sep 08 00:33:58.974490 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:58.974465 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-fgxt2" Sep 08 00:33:58.977015 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:58.976978 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qhpkm\"" Sep 08 00:33:58.988036 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:58.988012 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-fgxt2"] Sep 08 00:33:58.988134 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:58.988055 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-9nb9x"] Sep 08 00:33:58.988134 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:58.988124 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-9nb9x" Sep 08 00:33:58.994068 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:58.994047 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 08 00:33:58.994187 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:58.994175 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 08 00:33:58.994273 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:58.994257 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 08 00:33:58.995128 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:58.995105 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tprc6\"" Sep 08 00:33:58.995234 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:58.995183 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 08 00:33:58.998419 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:58.998400 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 08 00:33:59.050516 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.050481 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2e160ae-d75d-4959-a5ab-d238b4113736-config\") pod \"console-operator-85b454874b-9nb9x\" (UID: \"a2e160ae-d75d-4959-a5ab-d238b4113736\") " pod="openshift-console-operator/console-operator-85b454874b-9nb9x" Sep 08 00:33:59.050516 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.050512 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2e160ae-d75d-4959-a5ab-d238b4113736-trusted-ca\") pod \"console-operator-85b454874b-9nb9x\" (UID: \"a2e160ae-d75d-4959-a5ab-d238b4113736\") " pod="openshift-console-operator/console-operator-85b454874b-9nb9x" Sep 08 00:33:59.050682 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.050532 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rp8l\" (UniqueName: \"kubernetes.io/projected/1be18960-3ddf-4f17-9e6e-65344ce2bbb1-kube-api-access-8rp8l\") pod \"network-check-source-685456655f-fgxt2\" (UID: \"1be18960-3ddf-4f17-9e6e-65344ce2bbb1\") " pod="openshift-network-diagnostics/network-check-source-685456655f-fgxt2" Sep 08 00:33:59.050682 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.050617 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2e160ae-d75d-4959-a5ab-d238b4113736-serving-cert\") pod \"console-operator-85b454874b-9nb9x\" (UID: \"a2e160ae-d75d-4959-a5ab-d238b4113736\") " pod="openshift-console-operator/console-operator-85b454874b-9nb9x" Sep 08 00:33:59.050751 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.050676 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnk62\" (UniqueName: \"kubernetes.io/projected/a2e160ae-d75d-4959-a5ab-d238b4113736-kube-api-access-qnk62\") pod \"console-operator-85b454874b-9nb9x\" (UID: \"a2e160ae-d75d-4959-a5ab-d238b4113736\") " pod="openshift-console-operator/console-operator-85b454874b-9nb9x" Sep 08 00:33:59.061964 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.061941 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwkws"] Sep 08 00:33:59.082794 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.082770 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8js6v"] Sep 08 00:33:59.082925 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.082897 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwkws" Sep 08 00:33:59.085376 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.085353 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qfscq\"" Sep 08 00:33:59.085493 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.085359 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 08 00:33:59.085493 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.085390 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 08 00:33:59.085614 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.085509 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 08 00:33:59.085871 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.085857 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 08 00:33:59.103112 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.103091 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwkws"] Sep 08 00:33:59.103231 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.103119 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8js6v"] Sep 08 00:33:59.103231 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.103193 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8js6v" Sep 08 00:33:59.105666 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.105644 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 08 00:33:59.105909 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.105894 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 08 00:33:59.106030 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.106015 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-plbf8\"" Sep 08 00:33:59.151944 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.151917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnk62\" (UniqueName: \"kubernetes.io/projected/a2e160ae-d75d-4959-a5ab-d238b4113736-kube-api-access-qnk62\") pod \"console-operator-85b454874b-9nb9x\" (UID: \"a2e160ae-d75d-4959-a5ab-d238b4113736\") " pod="openshift-console-operator/console-operator-85b454874b-9nb9x" Sep 08 00:33:59.152086 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.151960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2e160ae-d75d-4959-a5ab-d238b4113736-config\") pod \"console-operator-85b454874b-9nb9x\" (UID: \"a2e160ae-d75d-4959-a5ab-d238b4113736\") " pod="openshift-console-operator/console-operator-85b454874b-9nb9x" Sep 08 00:33:59.152086 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.151977 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2e160ae-d75d-4959-a5ab-d238b4113736-trusted-ca\") pod \"console-operator-85b454874b-9nb9x\" (UID: \"a2e160ae-d75d-4959-a5ab-d238b4113736\") " pod="openshift-console-operator/console-operator-85b454874b-9nb9x" Sep 08 00:33:59.152086 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.152013 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rp8l\" (UniqueName: \"kubernetes.io/projected/1be18960-3ddf-4f17-9e6e-65344ce2bbb1-kube-api-access-8rp8l\") pod \"network-check-source-685456655f-fgxt2\" (UID: \"1be18960-3ddf-4f17-9e6e-65344ce2bbb1\") " pod="openshift-network-diagnostics/network-check-source-685456655f-fgxt2" Sep 08 00:33:59.152086 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.152063 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vpd5\" (UniqueName: \"kubernetes.io/projected/2e449098-718c-4079-bc8d-cfd13c463184-kube-api-access-6vpd5\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pwkws\" (UID: \"2e449098-718c-4079-bc8d-cfd13c463184\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwkws" Sep 08 00:33:59.152276 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.152094 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e449098-718c-4079-bc8d-cfd13c463184-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pwkws\" (UID: \"2e449098-718c-4079-bc8d-cfd13c463184\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwkws" Sep 08 00:33:59.152276 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.152204 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2e449098-718c-4079-bc8d-cfd13c463184-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pwkws\" (UID: \"2e449098-718c-4079-bc8d-cfd13c463184\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwkws" Sep 08 00:33:59.152276 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.152267 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjc82\" (UniqueName: \"kubernetes.io/projected/c33f752b-afab-4bad-8bc7-7e32c7817904-kube-api-access-vjc82\") pod \"volume-data-source-validator-695c895bd6-8js6v\" (UID: \"c33f752b-afab-4bad-8bc7-7e32c7817904\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8js6v" Sep 08 00:33:59.152399 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.152353 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2e160ae-d75d-4959-a5ab-d238b4113736-serving-cert\") pod \"console-operator-85b454874b-9nb9x\" (UID: \"a2e160ae-d75d-4959-a5ab-d238b4113736\") " pod="openshift-console-operator/console-operator-85b454874b-9nb9x" Sep 08 00:33:59.152683 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.152667 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a2e160ae-d75d-4959-a5ab-d238b4113736-config\") pod \"console-operator-85b454874b-9nb9x\" (UID: \"a2e160ae-d75d-4959-a5ab-d238b4113736\") " pod="openshift-console-operator/console-operator-85b454874b-9nb9x" Sep 08 00:33:59.152911 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.152892 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2e160ae-d75d-4959-a5ab-d238b4113736-trusted-ca\") pod \"console-operator-85b454874b-9nb9x\" (UID: \"a2e160ae-d75d-4959-a5ab-d238b4113736\") " pod="openshift-console-operator/console-operator-85b454874b-9nb9x" Sep 08 00:33:59.155594 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.155574 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a2e160ae-d75d-4959-a5ab-d238b4113736-serving-cert\") pod \"console-operator-85b454874b-9nb9x\" (UID: \"a2e160ae-d75d-4959-a5ab-d238b4113736\") " pod="openshift-console-operator/console-operator-85b454874b-9nb9x" Sep 08 00:33:59.163072 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.163052 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnk62\" (UniqueName: \"kubernetes.io/projected/a2e160ae-d75d-4959-a5ab-d238b4113736-kube-api-access-qnk62\") pod \"console-operator-85b454874b-9nb9x\" (UID: \"a2e160ae-d75d-4959-a5ab-d238b4113736\") " pod="openshift-console-operator/console-operator-85b454874b-9nb9x" Sep 08 00:33:59.163501 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.163479 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rp8l\" (UniqueName: \"kubernetes.io/projected/1be18960-3ddf-4f17-9e6e-65344ce2bbb1-kube-api-access-8rp8l\") pod \"network-check-source-685456655f-fgxt2\" (UID: \"1be18960-3ddf-4f17-9e6e-65344ce2bbb1\") " pod="openshift-network-diagnostics/network-check-source-685456655f-fgxt2" Sep 08 00:33:59.167060 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.167038 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69b4f78f5c-25bvx"] Sep 08 00:33:59.192915 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.192895 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69b4f78f5c-25bvx"] Sep 08 00:33:59.193015 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.192975 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.195711 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.195691 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 08 00:33:59.195918 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.195899 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 08 00:33:59.195918 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.195911 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ksxqj\"" Sep 08 00:33:59.196081 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.195927 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 08 00:33:59.202922 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.202896 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 08 00:33:59.253417 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.253335 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d44ebf3e-e583-4c18-a180-6cc12562201f-ca-trust-extracted\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.253417 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.253371 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d44ebf3e-e583-4c18-a180-6cc12562201f-installation-pull-secrets\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.253417 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.253404 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vpd5\" (UniqueName: \"kubernetes.io/projected/2e449098-718c-4079-bc8d-cfd13c463184-kube-api-access-6vpd5\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pwkws\" (UID: \"2e449098-718c-4079-bc8d-cfd13c463184\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwkws" Sep 08 00:33:59.253647 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.253428 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-bound-sa-token\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.253647 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.253476 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d44ebf3e-e583-4c18-a180-6cc12562201f-image-registry-private-configuration\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.253647 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.253515 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e449098-718c-4079-bc8d-cfd13c463184-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pwkws\" (UID: \"2e449098-718c-4079-bc8d-cfd13c463184\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwkws" Sep 08 00:33:59.253647 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.253536 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2e449098-718c-4079-bc8d-cfd13c463184-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pwkws\" (UID: \"2e449098-718c-4079-bc8d-cfd13c463184\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwkws" Sep 08 00:33:59.253647 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.253594 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjc82\" (UniqueName: \"kubernetes.io/projected/c33f752b-afab-4bad-8bc7-7e32c7817904-kube-api-access-vjc82\") pod \"volume-data-source-validator-695c895bd6-8js6v\" (UID: \"c33f752b-afab-4bad-8bc7-7e32c7817904\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8js6v" Sep 08 00:33:59.253647 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.253624 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-certificates\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.254068 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.253701 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d44ebf3e-e583-4c18-a180-6cc12562201f-trusted-ca\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.254068 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.253733 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k25mf\" (UniqueName: \"kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-kube-api-access-k25mf\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.254068 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.253787 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-tls\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.254068 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.254057 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2e449098-718c-4079-bc8d-cfd13c463184-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pwkws\" (UID: \"2e449098-718c-4079-bc8d-cfd13c463184\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwkws" Sep 08 00:33:59.255757 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.255729 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2e449098-718c-4079-bc8d-cfd13c463184-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pwkws\" (UID: \"2e449098-718c-4079-bc8d-cfd13c463184\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwkws" Sep 08 00:33:59.268905 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.268879 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vpd5\" (UniqueName: \"kubernetes.io/projected/2e449098-718c-4079-bc8d-cfd13c463184-kube-api-access-6vpd5\") pod \"kube-storage-version-migrator-operator-8d7dd4675-pwkws\" (UID: \"2e449098-718c-4079-bc8d-cfd13c463184\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwkws" Sep 08 00:33:59.270091 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.270074 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjc82\" (UniqueName: \"kubernetes.io/projected/c33f752b-afab-4bad-8bc7-7e32c7817904-kube-api-access-vjc82\") pod \"volume-data-source-validator-695c895bd6-8js6v\" (UID: \"c33f752b-afab-4bad-8bc7-7e32c7817904\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8js6v" Sep 08 00:33:59.283875 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.283856 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-fgxt2" Sep 08 00:33:59.296085 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.296063 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-9nb9x" Sep 08 00:33:59.354448 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.354422 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d44ebf3e-e583-4c18-a180-6cc12562201f-trusted-ca\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.354547 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.354459 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k25mf\" (UniqueName: \"kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-kube-api-access-k25mf\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.354547 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.354485 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-tls\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.354547 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.354522 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d44ebf3e-e583-4c18-a180-6cc12562201f-ca-trust-extracted\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.354547 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.354537 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d44ebf3e-e583-4c18-a180-6cc12562201f-installation-pull-secrets\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.354751 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.354561 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-bound-sa-token\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.354751 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.354590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d44ebf3e-e583-4c18-a180-6cc12562201f-image-registry-private-configuration\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.354751 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.354630 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-certificates\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.354751 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:59.354692 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 00:33:59.354751 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:59.354716 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69b4f78f5c-25bvx: secret "image-registry-tls" not found Sep 08 00:33:59.355033 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:59.354789 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-tls podName:d44ebf3e-e583-4c18-a180-6cc12562201f nodeName:}" failed. No retries permitted until 2026-09-08 00:33:59.854767319 +0000 UTC m=+53.550008469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-tls") pod "image-registry-69b4f78f5c-25bvx" (UID: "d44ebf3e-e583-4c18-a180-6cc12562201f") : secret "image-registry-tls" not found Sep 08 00:33:59.355939 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.355889 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d44ebf3e-e583-4c18-a180-6cc12562201f-ca-trust-extracted\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.356093 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.356070 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d44ebf3e-e583-4c18-a180-6cc12562201f-trusted-ca\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.356177 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.356074 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-certificates\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.357094 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.357069 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d44ebf3e-e583-4c18-a180-6cc12562201f-installation-pull-secrets\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.357231 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.357210 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d44ebf3e-e583-4c18-a180-6cc12562201f-image-registry-private-configuration\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.363784 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.363755 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k25mf\" (UniqueName: \"kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-kube-api-access-k25mf\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.364188 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.364167 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-bound-sa-token\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.391381 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.391343 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwkws" Sep 08 00:33:59.411474 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.411443 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8js6v" Sep 08 00:33:59.567384 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.567354 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-fgxt2"] Sep 08 00:33:59.571194 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:59.571172 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be18960_3ddf_4f17_9e6e_65344ce2bbb1.slice/crio-4d29b9805490bda6c5369dc2535097c3c72ba9f7cc67339c8d12d77fa4662e2d WatchSource:0}: Error finding container 4d29b9805490bda6c5369dc2535097c3c72ba9f7cc67339c8d12d77fa4662e2d: Status 404 returned error can't find the container with id 4d29b9805490bda6c5369dc2535097c3c72ba9f7cc67339c8d12d77fa4662e2d Sep 08 00:33:59.608325 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.608291 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-9nb9x"] Sep 08 00:33:59.610934 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:59.610913 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e160ae_d75d_4959_a5ab_d238b4113736.slice/crio-f8e5d42c82494236386f1226bfab59cfd91d4677efe1f924537e16b4b653c883 WatchSource:0}: Error finding container f8e5d42c82494236386f1226bfab59cfd91d4677efe1f924537e16b4b653c883: Status 404 returned error can't find the container with id f8e5d42c82494236386f1226bfab59cfd91d4677efe1f924537e16b4b653c883 Sep 08 00:33:59.691692 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.691664 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwkws"] Sep 08 00:33:59.695022 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:59.694964 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e449098_718c_4079_bc8d_cfd13c463184.slice/crio-46a66ae6bd76aae0ee9d7516ae9d15bd9e5197943123f4f99d4c81c554d8a9a4 WatchSource:0}: Error finding container 46a66ae6bd76aae0ee9d7516ae9d15bd9e5197943123f4f99d4c81c554d8a9a4: Status 404 returned error can't find the container with id 46a66ae6bd76aae0ee9d7516ae9d15bd9e5197943123f4f99d4c81c554d8a9a4 Sep 08 00:33:59.710346 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.710323 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8js6v"] Sep 08 00:33:59.713326 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:33:59.713302 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33f752b_afab_4bad_8bc7_7e32c7817904.slice/crio-066c726c8964dd993bc3dfb5915cdb1cb425ef684d931805ee3eda8b700904f7 WatchSource:0}: Error finding container 066c726c8964dd993bc3dfb5915cdb1cb425ef684d931805ee3eda8b700904f7: Status 404 returned error can't find the container with id 066c726c8964dd993bc3dfb5915cdb1cb425ef684d931805ee3eda8b700904f7 Sep 08 00:33:59.859340 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:33:59.859258 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-tls\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:33:59.859474 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:59.859374 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 00:33:59.859474 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:59.859386 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69b4f78f5c-25bvx: secret "image-registry-tls" not found Sep 08 00:33:59.859474 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:33:59.859436 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-tls podName:d44ebf3e-e583-4c18-a180-6cc12562201f nodeName:}" failed. No retries permitted until 2026-09-08 00:34:00.859423205 +0000 UTC m=+54.554664354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-tls") pod "image-registry-69b4f78f5c-25bvx" (UID: "d44ebf3e-e583-4c18-a180-6cc12562201f") : secret "image-registry-tls" not found Sep 08 00:34:00.137415 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:00.137301 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-9nb9x" event={"ID":"a2e160ae-d75d-4959-a5ab-d238b4113736","Type":"ContainerStarted","Data":"f8e5d42c82494236386f1226bfab59cfd91d4677efe1f924537e16b4b653c883"} Sep 08 00:34:00.138651 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:00.138594 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-fgxt2" event={"ID":"1be18960-3ddf-4f17-9e6e-65344ce2bbb1","Type":"ContainerStarted","Data":"4d29b9805490bda6c5369dc2535097c3c72ba9f7cc67339c8d12d77fa4662e2d"} Sep 08 00:34:00.139855 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:00.139804 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8js6v" event={"ID":"c33f752b-afab-4bad-8bc7-7e32c7817904","Type":"ContainerStarted","Data":"066c726c8964dd993bc3dfb5915cdb1cb425ef684d931805ee3eda8b700904f7"} Sep 08 00:34:00.141171 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:00.141151 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwkws" event={"ID":"2e449098-718c-4079-bc8d-cfd13c463184","Type":"ContainerStarted","Data":"46a66ae6bd76aae0ee9d7516ae9d15bd9e5197943123f4f99d4c81c554d8a9a4"} Sep 08 00:34:00.869224 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:00.869185 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-tls\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:34:00.869465 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:00.869429 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 00:34:00.869465 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:00.869448 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69b4f78f5c-25bvx: secret "image-registry-tls" not found Sep 08 00:34:00.869577 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:00.869530 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-tls podName:d44ebf3e-e583-4c18-a180-6cc12562201f nodeName:}" failed. No retries permitted until 2026-09-08 00:34:02.869511228 +0000 UTC m=+56.564752372 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-tls") pod "image-registry-69b4f78f5c-25bvx" (UID: "d44ebf3e-e583-4c18-a180-6cc12562201f") : secret "image-registry-tls" not found Sep 08 00:34:02.887388 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:02.887346 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-tls\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:34:02.887802 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:02.887494 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 00:34:02.887802 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:02.887510 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69b4f78f5c-25bvx: secret "image-registry-tls" not found Sep 08 00:34:02.887802 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:02.887571 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-tls podName:d44ebf3e-e583-4c18-a180-6cc12562201f nodeName:}" failed. No retries permitted until 2026-09-08 00:34:06.887552964 +0000 UTC m=+60.582794117 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-tls") pod "image-registry-69b4f78f5c-25bvx" (UID: "d44ebf3e-e583-4c18-a180-6cc12562201f") : secret "image-registry-tls" not found Sep 08 00:34:03.371550 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:03.371519 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hzcjc"] Sep 08 00:34:03.373754 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:03.373739 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:03.379764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:03.379740 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 08 00:34:03.380044 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:03.380029 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 08 00:34:03.380096 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:03.380064 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 08 00:34:03.381092 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:03.381077 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xt6cx\"" Sep 08 00:34:03.382720 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:03.382706 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 08 00:34:03.392704 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:03.392681 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hzcjc"] Sep 08 00:34:03.493238 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:03.493137 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/20bb0c4e-2cd4-468f-88e8-979c012be938-data-volume\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:03.493238 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:03.493195 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/20bb0c4e-2cd4-468f-88e8-979c012be938-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:03.493468 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:03.493229 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/20bb0c4e-2cd4-468f-88e8-979c012be938-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:03.493468 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:03.493330 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/20bb0c4e-2cd4-468f-88e8-979c012be938-crio-socket\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:03.493468 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:03.493360 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74bbr\" (UniqueName: \"kubernetes.io/projected/20bb0c4e-2cd4-468f-88e8-979c012be938-kube-api-access-74bbr\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:03.593763 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:03.593728 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/20bb0c4e-2cd4-468f-88e8-979c012be938-crio-socket\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:03.593763 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:03.593765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74bbr\" (UniqueName: \"kubernetes.io/projected/20bb0c4e-2cd4-468f-88e8-979c012be938-kube-api-access-74bbr\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:03.593941 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:03.593853 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/20bb0c4e-2cd4-468f-88e8-979c012be938-crio-socket\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:03.593941 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:03.593873 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/20bb0c4e-2cd4-468f-88e8-979c012be938-data-volume\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:03.593941 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:03.593895 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/20bb0c4e-2cd4-468f-88e8-979c012be938-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:03.593941 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:03.593913 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/20bb0c4e-2cd4-468f-88e8-979c012be938-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:03.594110 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:03.594029 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 00:34:03.594110 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:03.594043 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 00:34:03.594110 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:03.594084 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20bb0c4e-2cd4-468f-88e8-979c012be938-insights-runtime-extractor-tls podName:20bb0c4e-2cd4-468f-88e8-979c012be938 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:04.094066935 +0000 UTC m=+57.789308076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/20bb0c4e-2cd4-468f-88e8-979c012be938-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzcjc" (UID: "20bb0c4e-2cd4-468f-88e8-979c012be938") : secret "insights-runtime-extractor-tls" not found Sep 08 00:34:03.594110 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:03.594101 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20bb0c4e-2cd4-468f-88e8-979c012be938-kube-rbac-proxy-cm podName:20bb0c4e-2cd4-468f-88e8-979c012be938 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:04.094092016 +0000 UTC m=+57.789333155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/20bb0c4e-2cd4-468f-88e8-979c012be938-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzcjc" (UID: "20bb0c4e-2cd4-468f-88e8-979c012be938") : configmap "kube-rbac-proxy" not found Sep 08 00:34:03.594282 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:03.594258 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/20bb0c4e-2cd4-468f-88e8-979c012be938-data-volume\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:03.605232 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:03.605200 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74bbr\" (UniqueName: \"kubernetes.io/projected/20bb0c4e-2cd4-468f-88e8-979c012be938-kube-api-access-74bbr\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:04.098646 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:04.097979 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/20bb0c4e-2cd4-468f-88e8-979c012be938-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:04.098646 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:04.098048 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/20bb0c4e-2cd4-468f-88e8-979c012be938-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:04.098646 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:04.098199 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 00:34:04.098646 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:04.098258 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20bb0c4e-2cd4-468f-88e8-979c012be938-kube-rbac-proxy-cm podName:20bb0c4e-2cd4-468f-88e8-979c012be938 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:05.098239996 +0000 UTC m=+58.793481147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/20bb0c4e-2cd4-468f-88e8-979c012be938-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzcjc" (UID: "20bb0c4e-2cd4-468f-88e8-979c012be938") : configmap "kube-rbac-proxy" not found Sep 08 00:34:04.098646 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:04.098655 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 00:34:04.099228 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:04.098708 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20bb0c4e-2cd4-468f-88e8-979c012be938-insights-runtime-extractor-tls podName:20bb0c4e-2cd4-468f-88e8-979c012be938 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:05.098693967 +0000 UTC m=+58.793935108 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/20bb0c4e-2cd4-468f-88e8-979c012be938-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzcjc" (UID: "20bb0c4e-2cd4-468f-88e8-979c012be938") : secret "insights-runtime-extractor-tls" not found Sep 08 00:34:04.153782 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:04.153745 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8js6v" event={"ID":"c33f752b-afab-4bad-8bc7-7e32c7817904","Type":"ContainerStarted","Data":"2ace68079f111860c8caf285e72ff9fe65da50b8a2ff7a02c9185f702bbc5227"} Sep 08 00:34:04.155279 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:04.155248 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwkws" event={"ID":"2e449098-718c-4079-bc8d-cfd13c463184","Type":"ContainerStarted","Data":"8f38f57ab1faa1819344daf2595078acb5449dd50c0890e8da14d7c0fd96edad"} Sep 08 00:34:04.156833 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:04.156816 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9nb9x_a2e160ae-d75d-4959-a5ab-d238b4113736/console-operator/0.log" Sep 08 00:34:04.156897 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:04.156847 2577 generic.go:356] "Generic (PLEG): container finished" podID="a2e160ae-d75d-4959-a5ab-d238b4113736" containerID="ac25fd28f82314b8f8bf711c5987f9f30342068e467600676998ab3f696558dc" exitCode=255 Sep 08 00:34:04.156897 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:04.156875 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-9nb9x" event={"ID":"a2e160ae-d75d-4959-a5ab-d238b4113736","Type":"ContainerDied","Data":"ac25fd28f82314b8f8bf711c5987f9f30342068e467600676998ab3f696558dc"} Sep 08 00:34:04.157130 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:04.157099 2577 scope.go:117] "RemoveContainer" containerID="ac25fd28f82314b8f8bf711c5987f9f30342068e467600676998ab3f696558dc" Sep 08 00:34:04.158538 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:04.158516 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-fgxt2" event={"ID":"1be18960-3ddf-4f17-9e6e-65344ce2bbb1","Type":"ContainerStarted","Data":"aeeafb34b4f6fc944f1d16e2f0d5762d6198dbe1a9433e2aa5515eaa15fa311d"} Sep 08 00:34:04.196068 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:04.196018 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-8js6v" podStartSLOduration=1.740345617 podStartE2EDuration="5.195987083s" totalImagesPullingTime="3.455641466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:04.174934264 +0000 UTC m=+57.870175424" watchObservedRunningTime="2026-09-08 00:34:04.195987083 +0000 UTC m=+57.891228235" Sep 08 00:34:04.219566 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:04.219516 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwkws" podStartSLOduration=1.7266369990000001 podStartE2EDuration="5.219496515s" totalImagesPullingTime="3.492859516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:04.219279094 +0000 UTC m=+57.914520264" watchObservedRunningTime="2026-09-08 00:34:04.219496515 +0000 UTC m=+57.914737674" Sep 08 00:34:04.259398 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:04.258937 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-fgxt2" podStartSLOduration=2.626627446 podStartE2EDuration="6.25892196s" totalImagesPullingTime="3.632294514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:04.257400408 +0000 UTC m=+57.952641808" watchObservedRunningTime="2026-09-08 00:34:04.25892196 +0000 UTC m=+57.954163115" Sep 08 00:34:05.106232 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:05.106184 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/20bb0c4e-2cd4-468f-88e8-979c012be938-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:05.106232 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:05.106228 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/20bb0c4e-2cd4-468f-88e8-979c012be938-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:05.106643 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:05.106340 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 00:34:05.106643 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:05.106356 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 00:34:05.106643 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:05.106402 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20bb0c4e-2cd4-468f-88e8-979c012be938-insights-runtime-extractor-tls podName:20bb0c4e-2cd4-468f-88e8-979c012be938 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:07.10638415 +0000 UTC m=+60.801625302 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/20bb0c4e-2cd4-468f-88e8-979c012be938-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzcjc" (UID: "20bb0c4e-2cd4-468f-88e8-979c012be938") : secret "insights-runtime-extractor-tls" not found Sep 08 00:34:05.106643 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:05.106417 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20bb0c4e-2cd4-468f-88e8-979c012be938-kube-rbac-proxy-cm podName:20bb0c4e-2cd4-468f-88e8-979c012be938 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:07.106410899 +0000 UTC m=+60.801652035 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/20bb0c4e-2cd4-468f-88e8-979c012be938-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzcjc" (UID: "20bb0c4e-2cd4-468f-88e8-979c012be938") : configmap "kube-rbac-proxy" not found Sep 08 00:34:05.163604 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:05.163577 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9nb9x_a2e160ae-d75d-4959-a5ab-d238b4113736/console-operator/1.log" Sep 08 00:34:05.164016 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:05.163984 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9nb9x_a2e160ae-d75d-4959-a5ab-d238b4113736/console-operator/0.log" Sep 08 00:34:05.164071 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:05.164034 2577 generic.go:356] "Generic (PLEG): container finished" podID="a2e160ae-d75d-4959-a5ab-d238b4113736" containerID="ecd8e973d48ed57d53acf47bba98d9cba27eaaa73b24a16229f150343dc25923" exitCode=255 Sep 08 00:34:05.164429 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:05.164366 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-9nb9x" event={"ID":"a2e160ae-d75d-4959-a5ab-d238b4113736","Type":"ContainerDied","Data":"ecd8e973d48ed57d53acf47bba98d9cba27eaaa73b24a16229f150343dc25923"} Sep 08 00:34:05.164636 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:05.164620 2577 scope.go:117] "RemoveContainer" containerID="ac25fd28f82314b8f8bf711c5987f9f30342068e467600676998ab3f696558dc" Sep 08 00:34:05.164850 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:05.164837 2577 scope.go:117] "RemoveContainer" containerID="ecd8e973d48ed57d53acf47bba98d9cba27eaaa73b24a16229f150343dc25923" Sep 08 00:34:05.165272 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:05.165241 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-9nb9x_openshift-console-operator(a2e160ae-d75d-4959-a5ab-d238b4113736)\"" pod="openshift-console-operator/console-operator-85b454874b-9nb9x" podUID="a2e160ae-d75d-4959-a5ab-d238b4113736" Sep 08 00:34:05.503427 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:05.503334 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zpwqj_2b13bec4-42f2-4fda-95df-9773dfd10505/dns-node-resolver/0.log" Sep 08 00:34:06.102914 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:06.102890 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-25zw7_3587d6fa-1292-4006-9971-9937f6de1058/node-ca/0.log" Sep 08 00:34:06.168189 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:06.168165 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9nb9x_a2e160ae-d75d-4959-a5ab-d238b4113736/console-operator/1.log" Sep 08 00:34:06.168555 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:06.168489 2577 scope.go:117] "RemoveContainer" containerID="ecd8e973d48ed57d53acf47bba98d9cba27eaaa73b24a16229f150343dc25923" Sep 08 00:34:06.168643 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:06.168624 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-9nb9x_openshift-console-operator(a2e160ae-d75d-4959-a5ab-d238b4113736)\"" pod="openshift-console-operator/console-operator-85b454874b-9nb9x" podUID="a2e160ae-d75d-4959-a5ab-d238b4113736" Sep 08 00:34:06.921907 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:06.921859 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-tls\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:34:06.922186 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:06.922044 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 00:34:06.922186 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:06.922069 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-69b4f78f5c-25bvx: secret "image-registry-tls" not found Sep 08 00:34:06.922186 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:06.922136 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-tls podName:d44ebf3e-e583-4c18-a180-6cc12562201f nodeName:}" failed. No retries permitted until 2026-09-08 00:34:14.922108964 +0000 UTC m=+68.617350115 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-tls") pod "image-registry-69b4f78f5c-25bvx" (UID: "d44ebf3e-e583-4c18-a180-6cc12562201f") : secret "image-registry-tls" not found Sep 08 00:34:07.122852 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:07.122811 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/20bb0c4e-2cd4-468f-88e8-979c012be938-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:07.123032 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:07.122915 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/20bb0c4e-2cd4-468f-88e8-979c012be938-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:07.123032 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:07.122936 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 00:34:07.123032 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:07.123009 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 00:34:07.123137 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:07.123014 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20bb0c4e-2cd4-468f-88e8-979c012be938-kube-rbac-proxy-cm podName:20bb0c4e-2cd4-468f-88e8-979c012be938 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:11.12298151 +0000 UTC m=+64.818222647 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/20bb0c4e-2cd4-468f-88e8-979c012be938-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzcjc" (UID: "20bb0c4e-2cd4-468f-88e8-979c012be938") : configmap "kube-rbac-proxy" not found Sep 08 00:34:07.123137 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:07.123059 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20bb0c4e-2cd4-468f-88e8-979c012be938-insights-runtime-extractor-tls podName:20bb0c4e-2cd4-468f-88e8-979c012be938 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:11.123048048 +0000 UTC m=+64.818289185 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/20bb0c4e-2cd4-468f-88e8-979c012be938-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzcjc" (UID: "20bb0c4e-2cd4-468f-88e8-979c012be938") : secret "insights-runtime-extractor-tls" not found Sep 08 00:34:07.304287 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:07.304264 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xvxpk_08d6201a-e98a-45e5-a580-8d3ab6ca3ef5/serve-healthcheck-canary/0.log" Sep 08 00:34:08.075816 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:08.075789 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dm6k" Sep 08 00:34:09.297013 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:09.296968 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-9nb9x" Sep 08 00:34:09.297390 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:09.297030 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-9nb9x" Sep 08 00:34:09.297390 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:09.297321 2577 scope.go:117] "RemoveContainer" containerID="ecd8e973d48ed57d53acf47bba98d9cba27eaaa73b24a16229f150343dc25923" Sep 08 00:34:09.297489 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:09.297470 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-9nb9x_openshift-console-operator(a2e160ae-d75d-4959-a5ab-d238b4113736)\"" pod="openshift-console-operator/console-operator-85b454874b-9nb9x" podUID="a2e160ae-d75d-4959-a5ab-d238b4113736" Sep 08 00:34:11.153948 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:11.153890 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/20bb0c4e-2cd4-468f-88e8-979c012be938-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:11.153948 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:11.153947 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/20bb0c4e-2cd4-468f-88e8-979c012be938-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:11.154506 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:11.154059 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 00:34:11.154506 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:11.154120 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20bb0c4e-2cd4-468f-88e8-979c012be938-insights-runtime-extractor-tls podName:20bb0c4e-2cd4-468f-88e8-979c012be938 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:19.154105276 +0000 UTC m=+72.849346413 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/20bb0c4e-2cd4-468f-88e8-979c012be938-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hzcjc" (UID: "20bb0c4e-2cd4-468f-88e8-979c012be938") : secret "insights-runtime-extractor-tls" not found Sep 08 00:34:11.154506 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:11.154064 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 00:34:11.154506 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:11.154218 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20bb0c4e-2cd4-468f-88e8-979c012be938-kube-rbac-proxy-cm podName:20bb0c4e-2cd4-468f-88e8-979c012be938 nodeName:}" failed. No retries permitted until 2026-09-08 00:34:19.154199976 +0000 UTC m=+72.849441116 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/20bb0c4e-2cd4-468f-88e8-979c012be938-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hzcjc" (UID: "20bb0c4e-2cd4-468f-88e8-979c012be938") : configmap "kube-rbac-proxy" not found Sep 08 00:34:12.463399 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:12.463345 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs\") pod \"network-metrics-daemon-5xvxv\" (UID: \"8d449e65-fe9f-4c43-9d0b-4e130446e7a7\") " pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:34:12.463789 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:12.463464 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret\") pod \"global-pull-secret-syncer-sl7cg\" (UID: \"d30a2b6e-de4c-4204-9427-08784f0330c8\") " pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:34:12.466630 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:12.466614 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 00:34:12.466901 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:12.466888 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 00:34:12.473860 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:12.473843 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 08 00:34:12.473913 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:12.473902 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs podName:8d449e65-fe9f-4c43-9d0b-4e130446e7a7 nodeName:}" failed. No retries permitted until 2026-09-08 00:35:16.473880216 +0000 UTC m=+130.169121353 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs") pod "network-metrics-daemon-5xvxv" (UID: "8d449e65-fe9f-4c43-9d0b-4e130446e7a7") : secret "metrics-daemon-secret" not found Sep 08 00:34:12.476144 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:12.476126 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d30a2b6e-de4c-4204-9427-08784f0330c8-original-pull-secret\") pod \"global-pull-secret-syncer-sl7cg\" (UID: \"d30a2b6e-de4c-4204-9427-08784f0330c8\") " pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:34:12.514306 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:12.514280 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sl7cg" Sep 08 00:34:12.664943 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:12.664901 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj2ft\" (UniqueName: \"kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft\") pod \"network-check-target-dfgn5\" (UID: \"012969b4-e601-4068-96e9-6f80acb76afb\") " pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:34:12.667508 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:12.667482 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj2ft\" (UniqueName: \"kubernetes.io/projected/012969b4-e601-4068-96e9-6f80acb76afb-kube-api-access-lj2ft\") pod \"network-check-target-dfgn5\" (UID: \"012969b4-e601-4068-96e9-6f80acb76afb\") " pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:34:12.812700 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:12.812673 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wspvj\"" Sep 08 00:34:12.818530 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:34:12.818495 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30a2b6e_de4c_4204_9427_08784f0330c8.slice/crio-03fb3673b6bc289c0c881c3dbf9b71502f42558f0901a87dac8b63a1ecb350c8 WatchSource:0}: Error finding container 03fb3673b6bc289c0c881c3dbf9b71502f42558f0901a87dac8b63a1ecb350c8: Status 404 returned error can't find the container with id 03fb3673b6bc289c0c881c3dbf9b71502f42558f0901a87dac8b63a1ecb350c8 Sep 08 00:34:12.819659 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:12.819639 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:34:12.821143 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:12.821114 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sl7cg"] Sep 08 00:34:12.967194 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:12.967165 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/840e6533-8a38-426a-bb9a-abc4cad33caa-metrics-tls\") pod \"dns-default-kfd8x\" (UID: \"840e6533-8a38-426a-bb9a-abc4cad33caa\") " pod="openshift-dns/dns-default-kfd8x" Sep 08 00:34:12.969247 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:12.969230 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/840e6533-8a38-426a-bb9a-abc4cad33caa-metrics-tls\") pod \"dns-default-kfd8x\" (UID: \"840e6533-8a38-426a-bb9a-abc4cad33caa\") " pod="openshift-dns/dns-default-kfd8x" Sep 08 00:34:13.093632 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:13.093547 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dfgn5"] Sep 08 00:34:13.096855 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:34:13.096825 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012969b4_e601_4068_96e9_6f80acb76afb.slice/crio-8a5137d964317f3838e962e4b094ef9a12deacb3e07379510147881bac1c4049 WatchSource:0}: Error finding container 8a5137d964317f3838e962e4b094ef9a12deacb3e07379510147881bac1c4049: Status 404 returned error can't find the container with id 8a5137d964317f3838e962e4b094ef9a12deacb3e07379510147881bac1c4049 Sep 08 00:34:13.185519 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:13.185487 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dfgn5" event={"ID":"012969b4-e601-4068-96e9-6f80acb76afb","Type":"ContainerStarted","Data":"8a5137d964317f3838e962e4b094ef9a12deacb3e07379510147881bac1c4049"} Sep 08 00:34:13.186547 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:13.186526 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sl7cg" event={"ID":"d30a2b6e-de4c-4204-9427-08784f0330c8","Type":"ContainerStarted","Data":"03fb3673b6bc289c0c881c3dbf9b71502f42558f0901a87dac8b63a1ecb350c8"} Sep 08 00:34:13.231550 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:13.231524 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bs5xs\"" Sep 08 00:34:13.239678 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:13.239655 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kfd8x" Sep 08 00:34:13.537210 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:13.537182 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kfd8x"] Sep 08 00:34:13.540946 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:34:13.540918 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod840e6533_8a38_426a_bb9a_abc4cad33caa.slice/crio-da0892a7ce4aa0c405e9f8d40cf4d0e42051d3d62a446ff0f9705c2f11b1fe8f WatchSource:0}: Error finding container da0892a7ce4aa0c405e9f8d40cf4d0e42051d3d62a446ff0f9705c2f11b1fe8f: Status 404 returned error can't find the container with id da0892a7ce4aa0c405e9f8d40cf4d0e42051d3d62a446ff0f9705c2f11b1fe8f Sep 08 00:34:14.192405 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:14.192370 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dfgn5" event={"ID":"012969b4-e601-4068-96e9-6f80acb76afb","Type":"ContainerStarted","Data":"fbfa069498661c2d3046712836a0f111f3c95e8a7aecb40549f70b45fe270db2"} Sep 08 00:34:14.192616 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:14.192465 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:34:14.193632 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:14.193604 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kfd8x" event={"ID":"840e6533-8a38-426a-bb9a-abc4cad33caa","Type":"ContainerStarted","Data":"da0892a7ce4aa0c405e9f8d40cf4d0e42051d3d62a446ff0f9705c2f11b1fe8f"} Sep 08 00:34:14.210801 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:14.210745 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dfgn5" podStartSLOduration=67.21072862 podStartE2EDuration="1m7.21072862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:14.20995663 +0000 UTC m=+67.905197790" watchObservedRunningTime="2026-09-08 00:34:14.21072862 +0000 UTC m=+67.905969782" Sep 08 00:34:14.986159 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:14.986124 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-tls\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:34:14.989155 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:14.989096 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-tls\") pod \"image-registry-69b4f78f5c-25bvx\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:34:15.107458 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:15.107419 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ksxqj\"" Sep 08 00:34:15.115202 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:15.115144 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:34:17.184198 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:17.184113 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69b4f78f5c-25bvx"] Sep 08 00:34:17.188211 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:34:17.188179 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44ebf3e_e583_4c18_a180_6cc12562201f.slice/crio-24166e8330097ac8d4c6959b98097cd3755f01e6acc97e66869ad75546d35d97 WatchSource:0}: Error finding container 24166e8330097ac8d4c6959b98097cd3755f01e6acc97e66869ad75546d35d97: Status 404 returned error can't find the container with id 24166e8330097ac8d4c6959b98097cd3755f01e6acc97e66869ad75546d35d97 Sep 08 00:34:17.205440 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:17.205410 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sl7cg" event={"ID":"d30a2b6e-de4c-4204-9427-08784f0330c8","Type":"ContainerStarted","Data":"d1d8ba949fc42028ced382864ca2c7e2a8b58eb5eefebb845620ffe1393d9981"} Sep 08 00:34:17.207236 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:17.207212 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kfd8x" event={"ID":"840e6533-8a38-426a-bb9a-abc4cad33caa","Type":"ContainerStarted","Data":"10cdccfac604e871672ed116545628e2cc4a8bd7a61c81dc98ea79c165768fce"} Sep 08 00:34:17.207339 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:17.207256 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kfd8x" event={"ID":"840e6533-8a38-426a-bb9a-abc4cad33caa","Type":"ContainerStarted","Data":"455944a9b225880fc9b4f3395db7e084ed61191b68b944512dc968d4c03db52b"} Sep 08 00:34:17.207339 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:17.207287 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kfd8x" Sep 08 00:34:17.208920 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:17.208871 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" event={"ID":"d44ebf3e-e583-4c18-a180-6cc12562201f","Type":"ContainerStarted","Data":"24166e8330097ac8d4c6959b98097cd3755f01e6acc97e66869ad75546d35d97"} Sep 08 00:34:17.220972 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:17.220927 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sl7cg" podStartSLOduration=66.11495199 podStartE2EDuration="1m10.220913593s" totalImagesPullingTime="4.105961603s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:17.220265129 +0000 UTC m=+70.915506289" watchObservedRunningTime="2026-09-08 00:34:17.220913593 +0000 UTC m=+70.916154751" Sep 08 00:34:17.236764 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:17.236695 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kfd8x" podStartSLOduration=32.865956443 podStartE2EDuration="36.23668089s" totalImagesPullingTime="3.370724447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:33:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:17.236285748 +0000 UTC m=+70.931526909" watchObservedRunningTime="2026-09-08 00:34:17.23668089 +0000 UTC m=+70.931922039" Sep 08 00:34:18.217678 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:18.217628 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" event={"ID":"d44ebf3e-e583-4c18-a180-6cc12562201f","Type":"ContainerStarted","Data":"0e28285993bde28d3049822d8451c0cb33d07ef1cfbb354f68950ae07ea43746"} Sep 08 00:34:18.218173 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:18.217897 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:34:18.238711 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:18.238668 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" podStartSLOduration=19.238655454 podStartE2EDuration="19.238655454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:18.237920312 +0000 UTC m=+71.933161496" watchObservedRunningTime="2026-09-08 00:34:18.238655454 +0000 UTC m=+71.933896613" Sep 08 00:34:19.217040 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:19.216983 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/20bb0c4e-2cd4-468f-88e8-979c012be938-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:19.217216 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:19.217047 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/20bb0c4e-2cd4-468f-88e8-979c012be938-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:19.217606 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:19.217577 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/20bb0c4e-2cd4-468f-88e8-979c012be938-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:19.219433 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:19.219410 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/20bb0c4e-2cd4-468f-88e8-979c012be938-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hzcjc\" (UID: \"20bb0c4e-2cd4-468f-88e8-979c012be938\") " pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:19.310405 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:19.310370 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xt6cx\"" Sep 08 00:34:19.317832 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:19.317810 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hzcjc" Sep 08 00:34:19.594094 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:19.594063 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hzcjc"] Sep 08 00:34:19.596967 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:34:19.596939 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20bb0c4e_2cd4_468f_88e8_979c012be938.slice/crio-eceb5293b5bba577e956f8287ebabba86a43874f7bfb799a20ade6b5dbcdd681 WatchSource:0}: Error finding container eceb5293b5bba577e956f8287ebabba86a43874f7bfb799a20ade6b5dbcdd681: Status 404 returned error can't find the container with id eceb5293b5bba577e956f8287ebabba86a43874f7bfb799a20ade6b5dbcdd681 Sep 08 00:34:20.225244 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:20.225200 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzcjc" event={"ID":"20bb0c4e-2cd4-468f-88e8-979c012be938","Type":"ContainerStarted","Data":"94b07279883327004a7c27aca204d66421128336cb8fa82287a6c51822a58024"} Sep 08 00:34:20.225702 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:20.225265 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzcjc" event={"ID":"20bb0c4e-2cd4-468f-88e8-979c012be938","Type":"ContainerStarted","Data":"eceb5293b5bba577e956f8287ebabba86a43874f7bfb799a20ade6b5dbcdd681"} Sep 08 00:34:21.231015 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:21.230962 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzcjc" event={"ID":"20bb0c4e-2cd4-468f-88e8-979c012be938","Type":"ContainerStarted","Data":"b9f4e71ae8135e84c2980b3791a3f77f070cd0b126f482c0ae33ee1f24da4c20"} Sep 08 00:34:22.235518 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:22.235441 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hzcjc" event={"ID":"20bb0c4e-2cd4-468f-88e8-979c012be938","Type":"ContainerStarted","Data":"3724cf5827bba0addd78c06e923022217767ac3c18148422547b829e36317b3c"} Sep 08 00:34:22.263487 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:22.263442 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hzcjc" podStartSLOduration=17.142374148 podStartE2EDuration="19.263426755s" totalImagesPullingTime="2.121052607s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 00:34:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:22.26317491 +0000 UTC m=+75.958416070" watchObservedRunningTime="2026-09-08 00:34:22.263426755 +0000 UTC m=+75.958667917" Sep 08 00:34:22.788862 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:22.788836 2577 scope.go:117] "RemoveContainer" containerID="ecd8e973d48ed57d53acf47bba98d9cba27eaaa73b24a16229f150343dc25923" Sep 08 00:34:23.240279 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:23.240200 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9nb9x_a2e160ae-d75d-4959-a5ab-d238b4113736/console-operator/1.log" Sep 08 00:34:23.240642 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:23.240322 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-9nb9x" event={"ID":"a2e160ae-d75d-4959-a5ab-d238b4113736","Type":"ContainerStarted","Data":"23e071a23a8d13d28020af19d3c9ba32db6e368eb3e973e72182402fd5b2b007"} Sep 08 00:34:23.240826 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:23.240808 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-9nb9x" Sep 08 00:34:23.259545 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:23.259501 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-9nb9x" podStartSLOduration=21.696997643 podStartE2EDuration="25.259488052s" totalImagesPullingTime="3.562490409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:23.258902471 +0000 UTC m=+76.954143630" watchObservedRunningTime="2026-09-08 00:34:23.259488052 +0000 UTC m=+76.954729210" Sep 08 00:34:23.266196 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:23.266176 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-9nb9x" Sep 08 00:34:27.219970 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:27.219942 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kfd8x" Sep 08 00:34:29.506887 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.506778 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc"] Sep 08 00:34:29.511297 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.511278 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" Sep 08 00:34:29.514869 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.514847 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 08 00:34:29.515035 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.514877 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 08 00:34:29.515035 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.514842 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 08 00:34:29.515035 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.514984 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 08 00:34:29.515237 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.515007 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 08 00:34:29.515237 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.515163 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 08 00:34:29.515356 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.515256 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 08 00:34:29.526127 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.526110 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc"] Sep 08 00:34:29.545273 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.545255 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69b4f78f5c-25bvx"] Sep 08 00:34:29.591460 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.591426 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c83c073d-f99c-4607-a622-09b8388e9789-ca\") pod \"cluster-proxy-proxy-agent-58d65bf64b-hrpmc\" (UID: \"c83c073d-f99c-4607-a622-09b8388e9789\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" Sep 08 00:34:29.591624 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.591465 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c83c073d-f99c-4607-a622-09b8388e9789-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58d65bf64b-hrpmc\" (UID: \"c83c073d-f99c-4607-a622-09b8388e9789\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" Sep 08 00:34:29.591624 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.591497 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwbs6\" (UniqueName: \"kubernetes.io/projected/c83c073d-f99c-4607-a622-09b8388e9789-kube-api-access-gwbs6\") pod \"cluster-proxy-proxy-agent-58d65bf64b-hrpmc\" (UID: \"c83c073d-f99c-4607-a622-09b8388e9789\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" Sep 08 00:34:29.591624 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.591575 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c83c073d-f99c-4607-a622-09b8388e9789-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58d65bf64b-hrpmc\" (UID: \"c83c073d-f99c-4607-a622-09b8388e9789\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" Sep 08 00:34:29.591766 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.591643 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c83c073d-f99c-4607-a622-09b8388e9789-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58d65bf64b-hrpmc\" (UID: \"c83c073d-f99c-4607-a622-09b8388e9789\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" Sep 08 00:34:29.591766 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.591670 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c83c073d-f99c-4607-a622-09b8388e9789-hub\") pod \"cluster-proxy-proxy-agent-58d65bf64b-hrpmc\" (UID: \"c83c073d-f99c-4607-a622-09b8388e9789\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" Sep 08 00:34:29.692036 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.691977 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c83c073d-f99c-4607-a622-09b8388e9789-ca\") pod \"cluster-proxy-proxy-agent-58d65bf64b-hrpmc\" (UID: \"c83c073d-f99c-4607-a622-09b8388e9789\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" Sep 08 00:34:29.692036 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.692040 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c83c073d-f99c-4607-a622-09b8388e9789-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58d65bf64b-hrpmc\" (UID: \"c83c073d-f99c-4607-a622-09b8388e9789\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" Sep 08 00:34:29.692295 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.692071 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwbs6\" (UniqueName: \"kubernetes.io/projected/c83c073d-f99c-4607-a622-09b8388e9789-kube-api-access-gwbs6\") pod \"cluster-proxy-proxy-agent-58d65bf64b-hrpmc\" (UID: \"c83c073d-f99c-4607-a622-09b8388e9789\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" Sep 08 00:34:29.692295 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.692090 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c83c073d-f99c-4607-a622-09b8388e9789-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58d65bf64b-hrpmc\" (UID: \"c83c073d-f99c-4607-a622-09b8388e9789\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" Sep 08 00:34:29.692295 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.692117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c83c073d-f99c-4607-a622-09b8388e9789-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58d65bf64b-hrpmc\" (UID: \"c83c073d-f99c-4607-a622-09b8388e9789\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" Sep 08 00:34:29.692295 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.692139 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c83c073d-f99c-4607-a622-09b8388e9789-hub\") pod \"cluster-proxy-proxy-agent-58d65bf64b-hrpmc\" (UID: \"c83c073d-f99c-4607-a622-09b8388e9789\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" Sep 08 00:34:29.692872 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.692838 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c83c073d-f99c-4607-a622-09b8388e9789-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58d65bf64b-hrpmc\" (UID: \"c83c073d-f99c-4607-a622-09b8388e9789\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" Sep 08 00:34:29.694611 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.694579 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c83c073d-f99c-4607-a622-09b8388e9789-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58d65bf64b-hrpmc\" (UID: \"c83c073d-f99c-4607-a622-09b8388e9789\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" Sep 08 00:34:29.694687 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.694582 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c83c073d-f99c-4607-a622-09b8388e9789-hub\") pod \"cluster-proxy-proxy-agent-58d65bf64b-hrpmc\" (UID: \"c83c073d-f99c-4607-a622-09b8388e9789\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" Sep 08 00:34:29.694934 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.694919 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c83c073d-f99c-4607-a622-09b8388e9789-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58d65bf64b-hrpmc\" (UID: \"c83c073d-f99c-4607-a622-09b8388e9789\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" Sep 08 00:34:29.694975 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.694944 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c83c073d-f99c-4607-a622-09b8388e9789-ca\") pod \"cluster-proxy-proxy-agent-58d65bf64b-hrpmc\" (UID: \"c83c073d-f99c-4607-a622-09b8388e9789\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" Sep 08 00:34:29.709256 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.709230 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwbs6\" (UniqueName: \"kubernetes.io/projected/c83c073d-f99c-4607-a622-09b8388e9789-kube-api-access-gwbs6\") pod \"cluster-proxy-proxy-agent-58d65bf64b-hrpmc\" (UID: \"c83c073d-f99c-4607-a622-09b8388e9789\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" Sep 08 00:34:29.829800 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:29.829767 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" Sep 08 00:34:30.107901 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:30.107772 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc"] Sep 08 00:34:30.110341 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:34:30.110313 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc83c073d_f99c_4607_a622_09b8388e9789.slice/crio-9263d7b07f90b667169d563fc6b158bff50f7374a2ae50a71abded87392e4875 WatchSource:0}: Error finding container 9263d7b07f90b667169d563fc6b158bff50f7374a2ae50a71abded87392e4875: Status 404 returned error can't find the container with id 9263d7b07f90b667169d563fc6b158bff50f7374a2ae50a71abded87392e4875 Sep 08 00:34:30.265301 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:30.265272 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" event={"ID":"c83c073d-f99c-4607-a622-09b8388e9789","Type":"ContainerStarted","Data":"9263d7b07f90b667169d563fc6b158bff50f7374a2ae50a71abded87392e4875"} Sep 08 00:34:38.129792 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.129761 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kk8cg"] Sep 08 00:34:38.149855 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.149829 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb"] Sep 08 00:34:38.149983 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.149965 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.153423 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.153398 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 08 00:34:38.153562 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.153508 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 08 00:34:38.153562 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.153523 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 08 00:34:38.153671 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.153556 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 08 00:34:38.153815 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.153802 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 08 00:34:38.154630 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.154610 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 08 00:34:38.154740 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.154615 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5q7dp\"" Sep 08 00:34:38.176208 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.176184 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb"] Sep 08 00:34:38.176306 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.176289 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" Sep 08 00:34:38.178758 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.178738 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 08 00:34:38.179059 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.179013 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 08 00:34:38.179170 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.179150 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-h2wrd\"" Sep 08 00:34:38.179431 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.179418 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 08 00:34:38.261245 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.261218 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77ca624-d279-42c8-a358-70b8ef77f29f-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-hbxnb\" (UID: \"c77ca624-d279-42c8-a358-70b8ef77f29f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" Sep 08 00:34:38.261353 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.261246 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/66e32f1b-8691-42b3-8852-835aec07fb6e-root\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.261353 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.261270 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c77ca624-d279-42c8-a358-70b8ef77f29f-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-hbxnb\" (UID: \"c77ca624-d279-42c8-a358-70b8ef77f29f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" Sep 08 00:34:38.261353 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.261290 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c77ca624-d279-42c8-a358-70b8ef77f29f-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-hbxnb\" (UID: \"c77ca624-d279-42c8-a358-70b8ef77f29f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" Sep 08 00:34:38.261353 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.261318 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c77ca624-d279-42c8-a358-70b8ef77f29f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-hbxnb\" (UID: \"c77ca624-d279-42c8-a358-70b8ef77f29f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" Sep 08 00:34:38.261353 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.261339 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c77ca624-d279-42c8-a358-70b8ef77f29f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-hbxnb\" (UID: \"c77ca624-d279-42c8-a358-70b8ef77f29f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" Sep 08 00:34:38.261353 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.261354 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn8xp\" (UniqueName: \"kubernetes.io/projected/66e32f1b-8691-42b3-8852-835aec07fb6e-kube-api-access-kn8xp\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.261550 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.261371 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/66e32f1b-8691-42b3-8852-835aec07fb6e-node-exporter-tls\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.261550 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.261395 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66e32f1b-8691-42b3-8852-835aec07fb6e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.261550 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.261458 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl7fh\" (UniqueName: \"kubernetes.io/projected/c77ca624-d279-42c8-a358-70b8ef77f29f-kube-api-access-zl7fh\") pod \"kube-state-metrics-74cb57df7-hbxnb\" (UID: \"c77ca624-d279-42c8-a358-70b8ef77f29f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" Sep 08 00:34:38.261550 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.261497 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66e32f1b-8691-42b3-8852-835aec07fb6e-sys\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.261550 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.261519 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/66e32f1b-8691-42b3-8852-835aec07fb6e-node-exporter-wtmp\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.261550 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.261546 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/66e32f1b-8691-42b3-8852-835aec07fb6e-node-exporter-accelerators-collector-config\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.261723 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.261579 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/66e32f1b-8691-42b3-8852-835aec07fb6e-node-exporter-textfile\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.261723 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.261601 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66e32f1b-8691-42b3-8852-835aec07fb6e-metrics-client-ca\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.291817 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.291791 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" event={"ID":"c83c073d-f99c-4607-a622-09b8388e9789","Type":"ContainerStarted","Data":"01dcb9fddb2420b3a91a194c263338fbdee55eb2717bbd783738bbb7bd233330"} Sep 08 00:34:38.291929 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.291833 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" event={"ID":"c83c073d-f99c-4607-a622-09b8388e9789","Type":"ContainerStarted","Data":"9d7737485c00a93f97f87e602ceb489a06190b59ddc8b65f3de77af04176325a"} Sep 08 00:34:38.362522 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.362487 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl7fh\" (UniqueName: \"kubernetes.io/projected/c77ca624-d279-42c8-a358-70b8ef77f29f-kube-api-access-zl7fh\") pod \"kube-state-metrics-74cb57df7-hbxnb\" (UID: \"c77ca624-d279-42c8-a358-70b8ef77f29f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" Sep 08 00:34:38.362674 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.362531 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66e32f1b-8691-42b3-8852-835aec07fb6e-sys\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.362674 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.362551 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/66e32f1b-8691-42b3-8852-835aec07fb6e-node-exporter-wtmp\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.362674 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.362607 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66e32f1b-8691-42b3-8852-835aec07fb6e-sys\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.362674 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.362648 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/66e32f1b-8691-42b3-8852-835aec07fb6e-node-exporter-accelerators-collector-config\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.362674 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.362662 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/66e32f1b-8691-42b3-8852-835aec07fb6e-node-exporter-wtmp\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.362881 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.362695 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/66e32f1b-8691-42b3-8852-835aec07fb6e-node-exporter-textfile\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.362881 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.362720 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66e32f1b-8691-42b3-8852-835aec07fb6e-metrics-client-ca\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.362881 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.362744 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77ca624-d279-42c8-a358-70b8ef77f29f-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-hbxnb\" (UID: \"c77ca624-d279-42c8-a358-70b8ef77f29f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" Sep 08 00:34:38.362881 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.362765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/66e32f1b-8691-42b3-8852-835aec07fb6e-root\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.362881 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.362807 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c77ca624-d279-42c8-a358-70b8ef77f29f-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-hbxnb\" (UID: \"c77ca624-d279-42c8-a358-70b8ef77f29f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" Sep 08 00:34:38.362881 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.362830 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/66e32f1b-8691-42b3-8852-835aec07fb6e-root\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.362881 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.362832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c77ca624-d279-42c8-a358-70b8ef77f29f-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-hbxnb\" (UID: \"c77ca624-d279-42c8-a358-70b8ef77f29f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" Sep 08 00:34:38.363271 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.362888 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c77ca624-d279-42c8-a358-70b8ef77f29f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-hbxnb\" (UID: \"c77ca624-d279-42c8-a358-70b8ef77f29f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" Sep 08 00:34:38.363271 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.362919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c77ca624-d279-42c8-a358-70b8ef77f29f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-hbxnb\" (UID: \"c77ca624-d279-42c8-a358-70b8ef77f29f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" Sep 08 00:34:38.363271 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.362941 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn8xp\" (UniqueName: \"kubernetes.io/projected/66e32f1b-8691-42b3-8852-835aec07fb6e-kube-api-access-kn8xp\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.363271 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.362972 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/66e32f1b-8691-42b3-8852-835aec07fb6e-node-exporter-tls\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.363271 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.363009 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66e32f1b-8691-42b3-8852-835aec07fb6e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.363271 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.363095 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/66e32f1b-8691-42b3-8852-835aec07fb6e-node-exporter-textfile\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.363579 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.363307 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c77ca624-d279-42c8-a358-70b8ef77f29f-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-hbxnb\" (UID: \"c77ca624-d279-42c8-a358-70b8ef77f29f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" Sep 08 00:34:38.363579 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.363347 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/66e32f1b-8691-42b3-8852-835aec07fb6e-node-exporter-accelerators-collector-config\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.363579 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.363426 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66e32f1b-8691-42b3-8852-835aec07fb6e-metrics-client-ca\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.363579 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:38.363459 2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 08 00:34:38.363579 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:38.363532 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66e32f1b-8691-42b3-8852-835aec07fb6e-node-exporter-tls podName:66e32f1b-8691-42b3-8852-835aec07fb6e nodeName:}" failed. No retries permitted until 2026-09-08 00:34:38.863511365 +0000 UTC m=+92.558752505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/66e32f1b-8691-42b3-8852-835aec07fb6e-node-exporter-tls") pod "node-exporter-kk8cg" (UID: "66e32f1b-8691-42b3-8852-835aec07fb6e") : secret "node-exporter-tls" not found Sep 08 00:34:38.363834 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.363663 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c77ca624-d279-42c8-a358-70b8ef77f29f-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-hbxnb\" (UID: \"c77ca624-d279-42c8-a358-70b8ef77f29f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" Sep 08 00:34:38.363891 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.363873 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c77ca624-d279-42c8-a358-70b8ef77f29f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-hbxnb\" (UID: \"c77ca624-d279-42c8-a358-70b8ef77f29f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" Sep 08 00:34:38.365253 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.365234 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77ca624-d279-42c8-a358-70b8ef77f29f-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-hbxnb\" (UID: \"c77ca624-d279-42c8-a358-70b8ef77f29f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" Sep 08 00:34:38.365385 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.365327 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/66e32f1b-8691-42b3-8852-835aec07fb6e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.365528 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.365508 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c77ca624-d279-42c8-a358-70b8ef77f29f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-hbxnb\" (UID: \"c77ca624-d279-42c8-a358-70b8ef77f29f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" Sep 08 00:34:38.370989 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.370969 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl7fh\" (UniqueName: \"kubernetes.io/projected/c77ca624-d279-42c8-a358-70b8ef77f29f-kube-api-access-zl7fh\") pod \"kube-state-metrics-74cb57df7-hbxnb\" (UID: \"c77ca624-d279-42c8-a358-70b8ef77f29f\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" Sep 08 00:34:38.373715 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.373694 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn8xp\" (UniqueName: \"kubernetes.io/projected/66e32f1b-8691-42b3-8852-835aec07fb6e-kube-api-access-kn8xp\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.484723 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.484690 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" Sep 08 00:34:38.782426 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.782404 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb"] Sep 08 00:34:38.784973 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:34:38.784946 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77ca624_d279_42c8_a358_70b8ef77f29f.slice/crio-966c9c5f58c41ce3864460d9a7e27ec931d6c6f1f59e3f4116328452df6a1bfc WatchSource:0}: Error finding container 966c9c5f58c41ce3864460d9a7e27ec931d6c6f1f59e3f4116328452df6a1bfc: Status 404 returned error can't find the container with id 966c9c5f58c41ce3864460d9a7e27ec931d6c6f1f59e3f4116328452df6a1bfc Sep 08 00:34:38.867320 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.867286 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/66e32f1b-8691-42b3-8852-835aec07fb6e-node-exporter-tls\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:38.869442 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:38.869424 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/66e32f1b-8691-42b3-8852-835aec07fb6e-node-exporter-tls\") pod \"node-exporter-kk8cg\" (UID: \"66e32f1b-8691-42b3-8852-835aec07fb6e\") " pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:39.069879 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:39.069804 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kk8cg" Sep 08 00:34:39.079668 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:34:39.079638 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e32f1b_8691_42b3_8852_835aec07fb6e.slice/crio-626c30ef13dfd39559ea9f9dfdda8bc6e02828fbd8ce05441ade99be20013f9e WatchSource:0}: Error finding container 626c30ef13dfd39559ea9f9dfdda8bc6e02828fbd8ce05441ade99be20013f9e: Status 404 returned error can't find the container with id 626c30ef13dfd39559ea9f9dfdda8bc6e02828fbd8ce05441ade99be20013f9e Sep 08 00:34:39.295711 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:39.295671 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kk8cg" event={"ID":"66e32f1b-8691-42b3-8852-835aec07fb6e","Type":"ContainerStarted","Data":"626c30ef13dfd39559ea9f9dfdda8bc6e02828fbd8ce05441ade99be20013f9e"} Sep 08 00:34:39.296893 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:39.296865 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" event={"ID":"c77ca624-d279-42c8-a358-70b8ef77f29f","Type":"ContainerStarted","Data":"966c9c5f58c41ce3864460d9a7e27ec931d6c6f1f59e3f4116328452df6a1bfc"} Sep 08 00:34:39.299099 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:39.299074 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" event={"ID":"c83c073d-f99c-4607-a622-09b8388e9789","Type":"ContainerStarted","Data":"db609ec88823f34b69027837cd48ace6528510d7f2e2b4d755dfce64e03aa7a3"} Sep 08 00:34:39.318645 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:39.318602 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58d65bf64b-hrpmc" podStartSLOduration=2.451276692 podStartE2EDuration="10.318589648s" totalImagesPullingTime="7.867312956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:39.318388455 +0000 UTC m=+93.013629615" watchObservedRunningTime="2026-09-08 00:34:39.318589648 +0000 UTC m=+93.013830806" Sep 08 00:34:39.551353 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:39.551321 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:34:41.308309 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:41.308274 2577 generic.go:356] "Generic (PLEG): container finished" podID="66e32f1b-8691-42b3-8852-835aec07fb6e" containerID="d09cb53703f0d4afb828d00f53f7287353960b513e0eef832f504bad23972f43" exitCode=0 Sep 08 00:34:41.308778 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:41.308356 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kk8cg" event={"ID":"66e32f1b-8691-42b3-8852-835aec07fb6e","Type":"ContainerDied","Data":"d09cb53703f0d4afb828d00f53f7287353960b513e0eef832f504bad23972f43"} Sep 08 00:34:41.310658 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:41.310631 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" event={"ID":"c77ca624-d279-42c8-a358-70b8ef77f29f","Type":"ContainerStarted","Data":"b68250c0ec54d2223f04046568e0ab16f94762d329c6d598f5be6100272edd84"} Sep 08 00:34:41.310755 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:41.310684 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" event={"ID":"c77ca624-d279-42c8-a358-70b8ef77f29f","Type":"ContainerStarted","Data":"99676d724dd9f450074fd551ba17c98ad60358ff2fe912dae5bf5e532fdeeae9"} Sep 08 00:34:41.310755 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:41.310704 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" event={"ID":"c77ca624-d279-42c8-a358-70b8ef77f29f","Type":"ContainerStarted","Data":"06dbb1248baa31b0278f3a16b0b80ab18ef68076cc2f364ea2b9f626becaca2b"} Sep 08 00:34:42.315949 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:42.315913 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kk8cg" event={"ID":"66e32f1b-8691-42b3-8852-835aec07fb6e","Type":"ContainerStarted","Data":"e1989bd4f30ca882f3bba1adf02fa1cf9d24206830052db6e599dfdca5ac4392"} Sep 08 00:34:42.316404 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:42.315972 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kk8cg" event={"ID":"66e32f1b-8691-42b3-8852-835aec07fb6e","Type":"ContainerStarted","Data":"cd87a94164bd521c1c0ea0dbace86b0ff4134f88d5d07677345413b3187b21c5"} Sep 08 00:34:42.346588 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:42.346541 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-hbxnb" podStartSLOduration=2.88352902 podStartE2EDuration="4.346525867s" totalImagesPullingTime="1.462996847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:41.364311942 +0000 UTC m=+95.059553100" watchObservedRunningTime="2026-09-08 00:34:42.346525867 +0000 UTC m=+96.041767025" Sep 08 00:34:42.347043 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:42.347018 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kk8cg" podStartSLOduration=3.102091924 podStartE2EDuration="4.347010546s" totalImagesPullingTime="1.169521858s" totalInitContainerRuntime="75.396764ms" isStatefulPod=true podCreationTimestamp="2026-09-08 00:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:42.345150313 +0000 UTC m=+96.040391476" watchObservedRunningTime="2026-09-08 00:34:42.347010546 +0000 UTC m=+96.042251695" Sep 08 00:34:45.199255 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:45.199222 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dfgn5" Sep 08 00:34:45.728368 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:45.728334 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78dc6f67ff-q42gw"] Sep 08 00:34:45.730927 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:45.730909 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:45.738515 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:45.738483 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"oauth-serving-cert\" is forbidden: User \"system:node:ip-10-0-139-137.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-139-137.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" type="*v1.ConfigMap" Sep 08 00:34:45.738632 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:45.738484 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"trusted-ca-bundle\" is forbidden: User \"system:node:ip-10-0-139-137.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-139-137.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" type="*v1.ConfigMap" Sep 08 00:34:45.738632 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:45.738505 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-139-137.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-139-137.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Sep 08 00:34:45.738632 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:45.738521 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"console-serving-cert\" is forbidden: User \"system:node:ip-10-0-139-137.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-139-137.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"console-serving-cert\"" type="*v1.Secret" Sep 08 00:34:45.738632 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:45.738508 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"console-dockercfg-9ml86\" is forbidden: User \"system:node:ip-10-0-139-137.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-139-137.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"console-dockercfg-9ml86\"" type="*v1.Secret" Sep 08 00:34:45.738632 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:45.738556 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-139-137.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-139-137.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Sep 08 00:34:45.738827 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:45.738562 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"console-oauth-config\" is forbidden: User \"system:node:ip-10-0-139-137.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-139-137.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"console-oauth-config\"" type="*v1.Secret" Sep 08 00:34:45.738827 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:45.738555 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"console-78dc6f67ff-q42gw\" is forbidden: User \"system:node:ip-10-0-139-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-139-137.ec2.internal' and this object" podUID="71f7da9c-54f4-4ae2-b98b-bb78e3aa831e" pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:45.738827 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:45.738590 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"console-config\" is forbidden: User \"system:node:ip-10-0-139-137.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-139-137.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"console-config\"" type="*v1.ConfigMap" Sep 08 00:34:45.738827 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:45.738592 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"service-ca\" is forbidden: User \"system:node:ip-10-0-139-137.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-139-137.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"service-ca\"" type="*v1.ConfigMap" Sep 08 00:34:45.756548 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:45.756524 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78dc6f67ff-q42gw"] Sep 08 00:34:45.927170 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:45.927138 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-service-ca\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:45.927328 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:45.927189 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-trusted-ca-bundle\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:45.927328 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:45.927245 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67j4n\" (UniqueName: \"kubernetes.io/projected/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-kube-api-access-67j4n\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:45.927328 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:45.927306 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-console-oauth-config\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:45.927442 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:45.927336 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-console-serving-cert\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:45.927442 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:45.927356 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-console-config\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:45.927442 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:45.927373 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-oauth-serving-cert\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:46.028525 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:46.028475 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-service-ca\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:46.028772 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:46.028754 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-trusted-ca-bundle\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:46.028879 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:46.028866 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67j4n\" (UniqueName: \"kubernetes.io/projected/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-kube-api-access-67j4n\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:46.029025 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:46.028986 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-console-oauth-config\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:46.029151 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:46.029139 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-console-serving-cert\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:46.029240 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:46.029229 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-console-config\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:46.029341 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:46.029327 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-oauth-serving-cert\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:46.546727 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:46.546703 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 08 00:34:46.550290 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:46.550269 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-service-ca\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:46.607311 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:46.607283 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 08 00:34:46.610874 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:46.610856 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-console-config\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:46.710722 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:46.710694 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 08 00:34:46.720640 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:46.720618 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9ml86\"" Sep 08 00:34:46.723966 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:46.723940 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-console-oauth-config\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:46.749117 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:46.749092 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 08 00:34:47.030249 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:47.030223 2577 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: failed to sync secret cache: timed out waiting for the condition Sep 08 00:34:47.030249 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:47.030235 2577 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: failed to sync configmap cache: timed out waiting for the condition Sep 08 00:34:47.030422 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:47.030303 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-console-serving-cert podName:71f7da9c-54f4-4ae2-b98b-bb78e3aa831e nodeName:}" failed. No retries permitted until 2026-09-08 00:34:47.530286649 +0000 UTC m=+101.225527785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-console-serving-cert") pod "console-78dc6f67ff-q42gw" (UID: "71f7da9c-54f4-4ae2-b98b-bb78e3aa831e") : failed to sync secret cache: timed out waiting for the condition Sep 08 00:34:47.030422 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:47.030235 2577 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: failed to sync configmap cache: timed out waiting for the condition Sep 08 00:34:47.030422 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:47.030344 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-oauth-serving-cert podName:71f7da9c-54f4-4ae2-b98b-bb78e3aa831e nodeName:}" failed. No retries permitted until 2026-09-08 00:34:47.530326513 +0000 UTC m=+101.225567667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-oauth-serving-cert") pod "console-78dc6f67ff-q42gw" (UID: "71f7da9c-54f4-4ae2-b98b-bb78e3aa831e") : failed to sync configmap cache: timed out waiting for the condition Sep 08 00:34:47.030422 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:47.030369 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-trusted-ca-bundle podName:71f7da9c-54f4-4ae2-b98b-bb78e3aa831e nodeName:}" failed. No retries permitted until 2026-09-08 00:34:47.530356783 +0000 UTC m=+101.225597931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-trusted-ca-bundle") pod "console-78dc6f67ff-q42gw" (UID: "71f7da9c-54f4-4ae2-b98b-bb78e3aa831e") : failed to sync configmap cache: timed out waiting for the condition Sep 08 00:34:47.096957 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:47.096934 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 08 00:34:47.104650 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:47.104629 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 08 00:34:47.111649 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:47.111626 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67j4n\" (UniqueName: \"kubernetes.io/projected/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-kube-api-access-67j4n\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:47.154504 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:47.154477 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 08 00:34:47.172709 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:47.172687 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 08 00:34:47.540688 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:47.540644 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-trusted-ca-bundle\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:47.540856 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:47.540738 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-console-serving-cert\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:47.540856 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:47.540770 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-oauth-serving-cert\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:47.541559 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:47.541529 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-oauth-serving-cert\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:47.541670 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:47.541561 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-trusted-ca-bundle\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:47.543137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:47.543116 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-console-serving-cert\") pod \"console-78dc6f67ff-q42gw\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:47.840774 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:47.840689 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:34:48.129955 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:48.129867 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78dc6f67ff-q42gw"] Sep 08 00:34:48.132678 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:34:48.132651 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f7da9c_54f4_4ae2_b98b_bb78e3aa831e.slice/crio-56d330dd8b76f6b80a96540ff91980c026685f144c8cbcf7bca9568c42994782 WatchSource:0}: Error finding container 56d330dd8b76f6b80a96540ff91980c026685f144c8cbcf7bca9568c42994782: Status 404 returned error can't find the container with id 56d330dd8b76f6b80a96540ff91980c026685f144c8cbcf7bca9568c42994782 Sep 08 00:34:48.337273 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:48.337236 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78dc6f67ff-q42gw" event={"ID":"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e","Type":"ContainerStarted","Data":"56d330dd8b76f6b80a96540ff91980c026685f144c8cbcf7bca9568c42994782"} Sep 08 00:34:52.359176 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:52.359141 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78dc6f67ff-q42gw" event={"ID":"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e","Type":"ContainerStarted","Data":"49ec175888de9577f2efc550ac7632d900430abdaa4eff275b6e66f5ebfa52f6"} Sep 08 00:34:52.394122 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:52.394061 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78dc6f67ff-q42gw" podStartSLOduration=4.092389867 podStartE2EDuration="7.394040485s" totalImagesPullingTime="3.301650618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:52.393684993 +0000 UTC m=+106.088926151" watchObservedRunningTime="2026-09-08 00:34:52.394040485 +0000 UTC m=+106.089281646" Sep 08 00:34:54.564656 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.564613 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" podUID="d44ebf3e-e583-4c18-a180-6cc12562201f" containerName="registry" containerID="cri-o://0e28285993bde28d3049822d8451c0cb33d07ef1cfbb354f68950ae07ea43746" gracePeriod=30 Sep 08 00:34:54.798703 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.798678 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:34:54.896112 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.896042 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d44ebf3e-e583-4c18-a180-6cc12562201f-ca-trust-extracted\") pod \"d44ebf3e-e583-4c18-a180-6cc12562201f\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " Sep 08 00:34:54.896112 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.896090 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-bound-sa-token\") pod \"d44ebf3e-e583-4c18-a180-6cc12562201f\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " Sep 08 00:34:54.896320 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.896123 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d44ebf3e-e583-4c18-a180-6cc12562201f-installation-pull-secrets\") pod \"d44ebf3e-e583-4c18-a180-6cc12562201f\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " Sep 08 00:34:54.896320 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.896139 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-certificates\") pod \"d44ebf3e-e583-4c18-a180-6cc12562201f\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " Sep 08 00:34:54.896320 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.896164 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d44ebf3e-e583-4c18-a180-6cc12562201f-image-registry-private-configuration\") pod \"d44ebf3e-e583-4c18-a180-6cc12562201f\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " Sep 08 00:34:54.896320 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.896193 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k25mf\" (UniqueName: \"kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-kube-api-access-k25mf\") pod \"d44ebf3e-e583-4c18-a180-6cc12562201f\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " Sep 08 00:34:54.896320 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.896211 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d44ebf3e-e583-4c18-a180-6cc12562201f-trusted-ca\") pod \"d44ebf3e-e583-4c18-a180-6cc12562201f\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " Sep 08 00:34:54.896320 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.896236 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-tls\") pod \"d44ebf3e-e583-4c18-a180-6cc12562201f\" (UID: \"d44ebf3e-e583-4c18-a180-6cc12562201f\") " Sep 08 00:34:54.896720 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.896649 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d44ebf3e-e583-4c18-a180-6cc12562201f" (UID: "d44ebf3e-e583-4c18-a180-6cc12562201f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:34:54.897233 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.897205 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44ebf3e-e583-4c18-a180-6cc12562201f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d44ebf3e-e583-4c18-a180-6cc12562201f" (UID: "d44ebf3e-e583-4c18-a180-6cc12562201f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:34:54.898774 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.898737 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d44ebf3e-e583-4c18-a180-6cc12562201f" (UID: "d44ebf3e-e583-4c18-a180-6cc12562201f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:34:54.898774 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.898747 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d44ebf3e-e583-4c18-a180-6cc12562201f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d44ebf3e-e583-4c18-a180-6cc12562201f" (UID: "d44ebf3e-e583-4c18-a180-6cc12562201f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 00:34:54.898910 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.898805 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d44ebf3e-e583-4c18-a180-6cc12562201f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d44ebf3e-e583-4c18-a180-6cc12562201f" (UID: "d44ebf3e-e583-4c18-a180-6cc12562201f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 00:34:54.899061 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.899028 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-kube-api-access-k25mf" (OuterVolumeSpecName: "kube-api-access-k25mf") pod "d44ebf3e-e583-4c18-a180-6cc12562201f" (UID: "d44ebf3e-e583-4c18-a180-6cc12562201f"). InnerVolumeSpecName "kube-api-access-k25mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:34:54.899770 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.899752 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d44ebf3e-e583-4c18-a180-6cc12562201f" (UID: "d44ebf3e-e583-4c18-a180-6cc12562201f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:34:54.907761 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.907667 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d44ebf3e-e583-4c18-a180-6cc12562201f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d44ebf3e-e583-4c18-a180-6cc12562201f" (UID: "d44ebf3e-e583-4c18-a180-6cc12562201f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:34:54.996852 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.996820 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k25mf\" (UniqueName: \"kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-kube-api-access-k25mf\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:34:54.996852 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.996847 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d44ebf3e-e583-4c18-a180-6cc12562201f-trusted-ca\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:34:54.996852 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.996857 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-tls\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:34:54.997074 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.996865 2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d44ebf3e-e583-4c18-a180-6cc12562201f-ca-trust-extracted\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:34:54.997074 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.996873 2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d44ebf3e-e583-4c18-a180-6cc12562201f-bound-sa-token\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:34:54.997074 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.996881 2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d44ebf3e-e583-4c18-a180-6cc12562201f-installation-pull-secrets\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:34:54.997074 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.996890 2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d44ebf3e-e583-4c18-a180-6cc12562201f-registry-certificates\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:34:54.997074 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:54.996899 2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d44ebf3e-e583-4c18-a180-6cc12562201f-image-registry-private-configuration\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:34:55.371535 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:55.371502 2577 generic.go:356] "Generic (PLEG): container finished" podID="d44ebf3e-e583-4c18-a180-6cc12562201f" containerID="0e28285993bde28d3049822d8451c0cb33d07ef1cfbb354f68950ae07ea43746" exitCode=0 Sep 08 00:34:55.371694 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:55.371547 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" Sep 08 00:34:55.371694 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:55.371549 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" event={"ID":"d44ebf3e-e583-4c18-a180-6cc12562201f","Type":"ContainerDied","Data":"0e28285993bde28d3049822d8451c0cb33d07ef1cfbb354f68950ae07ea43746"} Sep 08 00:34:55.371694 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:55.371674 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69b4f78f5c-25bvx" event={"ID":"d44ebf3e-e583-4c18-a180-6cc12562201f","Type":"ContainerDied","Data":"24166e8330097ac8d4c6959b98097cd3755f01e6acc97e66869ad75546d35d97"} Sep 08 00:34:55.371792 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:55.371702 2577 scope.go:117] "RemoveContainer" containerID="0e28285993bde28d3049822d8451c0cb33d07ef1cfbb354f68950ae07ea43746" Sep 08 00:34:55.380901 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:55.380835 2577 scope.go:117] "RemoveContainer" containerID="0e28285993bde28d3049822d8451c0cb33d07ef1cfbb354f68950ae07ea43746" Sep 08 00:34:55.381363 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:34:55.381328 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e28285993bde28d3049822d8451c0cb33d07ef1cfbb354f68950ae07ea43746\": container with ID starting with 0e28285993bde28d3049822d8451c0cb33d07ef1cfbb354f68950ae07ea43746 not found: ID does not exist" containerID="0e28285993bde28d3049822d8451c0cb33d07ef1cfbb354f68950ae07ea43746" Sep 08 00:34:55.381482 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:55.381371 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e28285993bde28d3049822d8451c0cb33d07ef1cfbb354f68950ae07ea43746"} err="failed to get container status \"0e28285993bde28d3049822d8451c0cb33d07ef1cfbb354f68950ae07ea43746\": rpc error: code = NotFound desc = could not find container \"0e28285993bde28d3049822d8451c0cb33d07ef1cfbb354f68950ae07ea43746\": container with ID starting with 0e28285993bde28d3049822d8451c0cb33d07ef1cfbb354f68950ae07ea43746 not found: ID does not exist" Sep 08 00:34:55.395405 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:55.395380 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-69b4f78f5c-25bvx"] Sep 08 00:34:55.401573 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:55.401551 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-69b4f78f5c-25bvx"] Sep 08 00:34:56.006038 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.005986 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78dc6f67ff-q42gw"] Sep 08 00:34:56.055281 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.055252 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f8cd9b975-4wq2s"] Sep 08 00:34:56.055517 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.055505 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d44ebf3e-e583-4c18-a180-6cc12562201f" containerName="registry" Sep 08 00:34:56.055559 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.055519 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d44ebf3e-e583-4c18-a180-6cc12562201f" containerName="registry" Sep 08 00:34:56.055596 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.055578 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d44ebf3e-e583-4c18-a180-6cc12562201f" containerName="registry" Sep 08 00:34:56.058916 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.058885 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.069165 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.069144 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f8cd9b975-4wq2s"] Sep 08 00:34:56.105886 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.105858 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-service-ca\") pod \"console-f8cd9b975-4wq2s\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.106096 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.105908 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-trusted-ca-bundle\") pod \"console-f8cd9b975-4wq2s\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.106096 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.105990 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-oauth-serving-cert\") pod \"console-f8cd9b975-4wq2s\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.106096 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.106061 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83762f70-bf25-4284-a707-8e304a3890fe-console-oauth-config\") pod \"console-f8cd9b975-4wq2s\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.106096 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.106084 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-console-config\") pod \"console-f8cd9b975-4wq2s\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.106257 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.106103 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83762f70-bf25-4284-a707-8e304a3890fe-console-serving-cert\") pod \"console-f8cd9b975-4wq2s\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.106257 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.106161 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2xdj\" (UniqueName: \"kubernetes.io/projected/83762f70-bf25-4284-a707-8e304a3890fe-kube-api-access-f2xdj\") pod \"console-f8cd9b975-4wq2s\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.206937 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.206894 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-trusted-ca-bundle\") pod \"console-f8cd9b975-4wq2s\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.206937 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.206941 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-oauth-serving-cert\") pod \"console-f8cd9b975-4wq2s\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.207193 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.206983 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83762f70-bf25-4284-a707-8e304a3890fe-console-oauth-config\") pod \"console-f8cd9b975-4wq2s\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.207193 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.207027 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-console-config\") pod \"console-f8cd9b975-4wq2s\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.207193 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.207097 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83762f70-bf25-4284-a707-8e304a3890fe-console-serving-cert\") pod \"console-f8cd9b975-4wq2s\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.207193 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.207144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2xdj\" (UniqueName: \"kubernetes.io/projected/83762f70-bf25-4284-a707-8e304a3890fe-kube-api-access-f2xdj\") pod \"console-f8cd9b975-4wq2s\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.207193 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.207169 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-service-ca\") pod \"console-f8cd9b975-4wq2s\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.207852 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.207827 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-console-config\") pod \"console-f8cd9b975-4wq2s\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.207976 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.207849 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-oauth-serving-cert\") pod \"console-f8cd9b975-4wq2s\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.207976 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.207966 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-trusted-ca-bundle\") pod \"console-f8cd9b975-4wq2s\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.208131 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.208016 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-service-ca\") pod \"console-f8cd9b975-4wq2s\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.209399 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.209370 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83762f70-bf25-4284-a707-8e304a3890fe-console-oauth-config\") pod \"console-f8cd9b975-4wq2s\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.209565 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.209546 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83762f70-bf25-4284-a707-8e304a3890fe-console-serving-cert\") pod \"console-f8cd9b975-4wq2s\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.218044 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.218022 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2xdj\" (UniqueName: \"kubernetes.io/projected/83762f70-bf25-4284-a707-8e304a3890fe-kube-api-access-f2xdj\") pod \"console-f8cd9b975-4wq2s\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.368168 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.368133 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:34:56.636537 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.636513 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f8cd9b975-4wq2s"] Sep 08 00:34:56.638625 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:34:56.638599 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83762f70_bf25_4284_a707_8e304a3890fe.slice/crio-e36a53d7c3e83a7dfe5df8145ec4d98f0044d3bb760a8b0778b7d453dea26f0e WatchSource:0}: Error finding container e36a53d7c3e83a7dfe5df8145ec4d98f0044d3bb760a8b0778b7d453dea26f0e: Status 404 returned error can't find the container with id e36a53d7c3e83a7dfe5df8145ec4d98f0044d3bb760a8b0778b7d453dea26f0e Sep 08 00:34:56.789719 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:56.789684 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d44ebf3e-e583-4c18-a180-6cc12562201f" path="/var/lib/kubelet/pods/d44ebf3e-e583-4c18-a180-6cc12562201f/volumes" Sep 08 00:34:57.383881 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:57.383840 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8cd9b975-4wq2s" event={"ID":"83762f70-bf25-4284-a707-8e304a3890fe","Type":"ContainerStarted","Data":"c57ae4d321972db5d12611065aeb0d841a914e4bd44f603a1aac23cc1cfac75d"} Sep 08 00:34:57.384380 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:57.383917 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8cd9b975-4wq2s" event={"ID":"83762f70-bf25-4284-a707-8e304a3890fe","Type":"ContainerStarted","Data":"e36a53d7c3e83a7dfe5df8145ec4d98f0044d3bb760a8b0778b7d453dea26f0e"} Sep 08 00:34:57.841174 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:34:57.841133 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:35:06.369080 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:06.369031 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:35:06.369080 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:06.369097 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:35:06.373797 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:06.373774 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:35:06.395405 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:06.395211 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f8cd9b975-4wq2s" podStartSLOduration=10.39519202 podStartE2EDuration="10.39519202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:34:57.406197852 +0000 UTC m=+111.101439013" watchObservedRunningTime="2026-09-08 00:35:06.39519202 +0000 UTC m=+120.090433180" Sep 08 00:35:06.421258 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:06.421235 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:35:16.564802 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:16.564740 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs\") pod \"network-metrics-daemon-5xvxv\" (UID: \"8d449e65-fe9f-4c43-9d0b-4e130446e7a7\") " pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:35:16.567068 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:16.567049 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d449e65-fe9f-4c43-9d0b-4e130446e7a7-metrics-certs\") pod \"network-metrics-daemon-5xvxv\" (UID: \"8d449e65-fe9f-4c43-9d0b-4e130446e7a7\") " pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:35:16.705379 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:16.705348 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5xvx4\"" Sep 08 00:35:16.713442 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:16.713423 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5xvxv" Sep 08 00:35:16.984234 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:16.984164 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5xvxv"] Sep 08 00:35:16.987349 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:35:16.987326 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d449e65_fe9f_4c43_9d0b_4e130446e7a7.slice/crio-f927eea8f3625c78dd60dc080609ba5077d5fb75d7b4f5f514bae0cecb045b0e WatchSource:0}: Error finding container f927eea8f3625c78dd60dc080609ba5077d5fb75d7b4f5f514bae0cecb045b0e: Status 404 returned error can't find the container with id f927eea8f3625c78dd60dc080609ba5077d5fb75d7b4f5f514bae0cecb045b0e Sep 08 00:35:17.458968 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:17.458930 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5xvxv" event={"ID":"8d449e65-fe9f-4c43-9d0b-4e130446e7a7","Type":"ContainerStarted","Data":"f927eea8f3625c78dd60dc080609ba5077d5fb75d7b4f5f514bae0cecb045b0e"} Sep 08 00:35:18.463837 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:18.463798 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5xvxv" event={"ID":"8d449e65-fe9f-4c43-9d0b-4e130446e7a7","Type":"ContainerStarted","Data":"1622449b2bea5761ddc39a97ce66f890a725f0132eff01940d153f2ab51176ef"} Sep 08 00:35:18.464244 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:18.463866 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5xvxv" event={"ID":"8d449e65-fe9f-4c43-9d0b-4e130446e7a7","Type":"ContainerStarted","Data":"0d7f6c4a6e3843806dbba1a6f09cfedfa2f1823175bb971585bcbc157218a3b5"} Sep 08 00:35:18.483231 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:18.483185 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5xvxv" podStartSLOduration=130.619830644 podStartE2EDuration="2m11.483168961s" totalImagesPullingTime="863.338317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:35:18.4810283 +0000 UTC m=+132.176269460" watchObservedRunningTime="2026-09-08 00:35:18.483168961 +0000 UTC m=+132.178410120" Sep 08 00:35:21.025331 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.025286 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-78dc6f67ff-q42gw" podUID="71f7da9c-54f4-4ae2-b98b-bb78e3aa831e" containerName="console" containerID="cri-o://49ec175888de9577f2efc550ac7632d900430abdaa4eff275b6e66f5ebfa52f6" gracePeriod=15 Sep 08 00:35:21.270893 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.270868 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:35:21.402619 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.402581 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-console-oauth-config\") pod \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " Sep 08 00:35:21.402812 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.402628 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-service-ca\") pod \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " Sep 08 00:35:21.402812 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.402656 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67j4n\" (UniqueName: \"kubernetes.io/projected/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-kube-api-access-67j4n\") pod \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " Sep 08 00:35:21.402812 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.402679 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-console-config\") pod \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " Sep 08 00:35:21.402812 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.402705 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-console-serving-cert\") pod \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " Sep 08 00:35:21.402812 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.402722 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-oauth-serving-cert\") pod \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " Sep 08 00:35:21.402812 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.402765 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-trusted-ca-bundle\") pod \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\" (UID: \"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e\") " Sep 08 00:35:21.403151 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.403116 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-service-ca" (OuterVolumeSpecName: "service-ca") pod "71f7da9c-54f4-4ae2-b98b-bb78e3aa831e" (UID: "71f7da9c-54f4-4ae2-b98b-bb78e3aa831e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:35:21.403203 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.403185 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-console-config" (OuterVolumeSpecName: "console-config") pod "71f7da9c-54f4-4ae2-b98b-bb78e3aa831e" (UID: "71f7da9c-54f4-4ae2-b98b-bb78e3aa831e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:35:21.403270 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.403222 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "71f7da9c-54f4-4ae2-b98b-bb78e3aa831e" (UID: "71f7da9c-54f4-4ae2-b98b-bb78e3aa831e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:35:21.403270 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.403262 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "71f7da9c-54f4-4ae2-b98b-bb78e3aa831e" (UID: "71f7da9c-54f4-4ae2-b98b-bb78e3aa831e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:35:21.405044 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.405013 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "71f7da9c-54f4-4ae2-b98b-bb78e3aa831e" (UID: "71f7da9c-54f4-4ae2-b98b-bb78e3aa831e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 00:35:21.405262 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.405227 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "71f7da9c-54f4-4ae2-b98b-bb78e3aa831e" (UID: "71f7da9c-54f4-4ae2-b98b-bb78e3aa831e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 00:35:21.405334 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.405278 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-kube-api-access-67j4n" (OuterVolumeSpecName: "kube-api-access-67j4n") pod "71f7da9c-54f4-4ae2-b98b-bb78e3aa831e" (UID: "71f7da9c-54f4-4ae2-b98b-bb78e3aa831e"). InnerVolumeSpecName "kube-api-access-67j4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:35:21.475277 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.475247 2577 generic.go:356] "Generic (PLEG): container finished" podID="71f7da9c-54f4-4ae2-b98b-bb78e3aa831e" containerID="49ec175888de9577f2efc550ac7632d900430abdaa4eff275b6e66f5ebfa52f6" exitCode=0 Sep 08 00:35:21.475424 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.475301 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78dc6f67ff-q42gw" Sep 08 00:35:21.475424 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.475313 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78dc6f67ff-q42gw" event={"ID":"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e","Type":"ContainerDied","Data":"49ec175888de9577f2efc550ac7632d900430abdaa4eff275b6e66f5ebfa52f6"} Sep 08 00:35:21.475424 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.475376 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78dc6f67ff-q42gw" event={"ID":"71f7da9c-54f4-4ae2-b98b-bb78e3aa831e","Type":"ContainerDied","Data":"56d330dd8b76f6b80a96540ff91980c026685f144c8cbcf7bca9568c42994782"} Sep 08 00:35:21.475424 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.475397 2577 scope.go:117] "RemoveContainer" containerID="49ec175888de9577f2efc550ac7632d900430abdaa4eff275b6e66f5ebfa52f6" Sep 08 00:35:21.483571 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.483552 2577 scope.go:117] "RemoveContainer" containerID="49ec175888de9577f2efc550ac7632d900430abdaa4eff275b6e66f5ebfa52f6" Sep 08 00:35:21.483831 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:35:21.483813 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ec175888de9577f2efc550ac7632d900430abdaa4eff275b6e66f5ebfa52f6\": container with ID starting with 49ec175888de9577f2efc550ac7632d900430abdaa4eff275b6e66f5ebfa52f6 not found: ID does not exist" containerID="49ec175888de9577f2efc550ac7632d900430abdaa4eff275b6e66f5ebfa52f6" Sep 08 00:35:21.483885 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.483838 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ec175888de9577f2efc550ac7632d900430abdaa4eff275b6e66f5ebfa52f6"} err="failed to get container status \"49ec175888de9577f2efc550ac7632d900430abdaa4eff275b6e66f5ebfa52f6\": rpc error: code = NotFound desc = could not find container \"49ec175888de9577f2efc550ac7632d900430abdaa4eff275b6e66f5ebfa52f6\": container with ID starting with 49ec175888de9577f2efc550ac7632d900430abdaa4eff275b6e66f5ebfa52f6 not found: ID does not exist" Sep 08 00:35:21.496100 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.496075 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78dc6f67ff-q42gw"] Sep 08 00:35:21.500373 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.500354 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78dc6f67ff-q42gw"] Sep 08 00:35:21.503552 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.503535 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-console-oauth-config\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:35:21.503604 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.503555 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-service-ca\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:35:21.503604 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.503566 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67j4n\" (UniqueName: \"kubernetes.io/projected/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-kube-api-access-67j4n\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:35:21.503604 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.503574 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-console-config\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:35:21.503604 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.503583 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-console-serving-cert\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:35:21.503604 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.503591 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-oauth-serving-cert\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:35:21.503604 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:21.503599 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e-trusted-ca-bundle\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:35:22.789580 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:22.789548 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71f7da9c-54f4-4ae2-b98b-bb78e3aa831e" path="/var/lib/kubelet/pods/71f7da9c-54f4-4ae2-b98b-bb78e3aa831e/volumes" Sep 08 00:35:33.826101 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:35:33.826073 2577 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e449098_718c_4079_bc8d_cfd13c463184.slice/crio-conmon-8f38f57ab1faa1819344daf2595078acb5449dd50c0890e8da14d7c0fd96edad.scope\": RecentStats: unable to find data in memory cache]" Sep 08 00:35:33.826418 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:35:33.826177 2577 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e449098_718c_4079_bc8d_cfd13c463184.slice/crio-conmon-8f38f57ab1faa1819344daf2595078acb5449dd50c0890e8da14d7c0fd96edad.scope\": RecentStats: unable to find data in memory cache]" Sep 08 00:35:34.523906 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:34.523871 2577 generic.go:356] "Generic (PLEG): container finished" podID="2e449098-718c-4079-bc8d-cfd13c463184" containerID="8f38f57ab1faa1819344daf2595078acb5449dd50c0890e8da14d7c0fd96edad" exitCode=0 Sep 08 00:35:34.524096 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:34.523951 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwkws" event={"ID":"2e449098-718c-4079-bc8d-cfd13c463184","Type":"ContainerDied","Data":"8f38f57ab1faa1819344daf2595078acb5449dd50c0890e8da14d7c0fd96edad"} Sep 08 00:35:34.524347 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:34.524331 2577 scope.go:117] "RemoveContainer" containerID="8f38f57ab1faa1819344daf2595078acb5449dd50c0890e8da14d7c0fd96edad" Sep 08 00:35:35.528689 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:35:35.528658 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-pwkws" event={"ID":"2e449098-718c-4079-bc8d-cfd13c463184","Type":"ContainerStarted","Data":"75c5e64d8ac9c9b24434a2e1c528eac81963f846e6ac92a3ea3f1ac231c3b086"} Sep 08 00:36:10.957510 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:10.957475 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f8cd9b975-4wq2s"] Sep 08 00:36:35.977269 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:35.977223 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f8cd9b975-4wq2s" podUID="83762f70-bf25-4284-a707-8e304a3890fe" containerName="console" containerID="cri-o://c57ae4d321972db5d12611065aeb0d841a914e4bd44f603a1aac23cc1cfac75d" gracePeriod=15 Sep 08 00:36:36.221319 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.221293 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:36:36.335627 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.335581 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83762f70-bf25-4284-a707-8e304a3890fe-console-oauth-config\") pod \"83762f70-bf25-4284-a707-8e304a3890fe\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " Sep 08 00:36:36.335627 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.335636 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2xdj\" (UniqueName: \"kubernetes.io/projected/83762f70-bf25-4284-a707-8e304a3890fe-kube-api-access-f2xdj\") pod \"83762f70-bf25-4284-a707-8e304a3890fe\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " Sep 08 00:36:36.335868 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.335659 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-console-config\") pod \"83762f70-bf25-4284-a707-8e304a3890fe\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " Sep 08 00:36:36.335868 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.335681 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83762f70-bf25-4284-a707-8e304a3890fe-console-serving-cert\") pod \"83762f70-bf25-4284-a707-8e304a3890fe\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " Sep 08 00:36:36.335868 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.335710 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-oauth-serving-cert\") pod \"83762f70-bf25-4284-a707-8e304a3890fe\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " Sep 08 00:36:36.335868 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.335733 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-trusted-ca-bundle\") pod \"83762f70-bf25-4284-a707-8e304a3890fe\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " Sep 08 00:36:36.335868 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.335776 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-service-ca\") pod \"83762f70-bf25-4284-a707-8e304a3890fe\" (UID: \"83762f70-bf25-4284-a707-8e304a3890fe\") " Sep 08 00:36:36.336288 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.336160 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-console-config" (OuterVolumeSpecName: "console-config") pod "83762f70-bf25-4284-a707-8e304a3890fe" (UID: "83762f70-bf25-4284-a707-8e304a3890fe"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:36:36.336381 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.336220 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "83762f70-bf25-4284-a707-8e304a3890fe" (UID: "83762f70-bf25-4284-a707-8e304a3890fe"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:36:36.336381 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.336297 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-service-ca" (OuterVolumeSpecName: "service-ca") pod "83762f70-bf25-4284-a707-8e304a3890fe" (UID: "83762f70-bf25-4284-a707-8e304a3890fe"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:36:36.336469 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.336360 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "83762f70-bf25-4284-a707-8e304a3890fe" (UID: "83762f70-bf25-4284-a707-8e304a3890fe"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:36:36.337830 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.337807 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83762f70-bf25-4284-a707-8e304a3890fe-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "83762f70-bf25-4284-a707-8e304a3890fe" (UID: "83762f70-bf25-4284-a707-8e304a3890fe"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 00:36:36.337830 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.337822 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83762f70-bf25-4284-a707-8e304a3890fe-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "83762f70-bf25-4284-a707-8e304a3890fe" (UID: "83762f70-bf25-4284-a707-8e304a3890fe"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 00:36:36.337941 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.337872 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83762f70-bf25-4284-a707-8e304a3890fe-kube-api-access-f2xdj" (OuterVolumeSpecName: "kube-api-access-f2xdj") pod "83762f70-bf25-4284-a707-8e304a3890fe" (UID: "83762f70-bf25-4284-a707-8e304a3890fe"). InnerVolumeSpecName "kube-api-access-f2xdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:36:36.437469 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.437420 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-service-ca\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:36:36.437469 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.437464 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83762f70-bf25-4284-a707-8e304a3890fe-console-oauth-config\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:36:36.437469 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.437474 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2xdj\" (UniqueName: \"kubernetes.io/projected/83762f70-bf25-4284-a707-8e304a3890fe-kube-api-access-f2xdj\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:36:36.437469 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.437484 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-console-config\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:36:36.437728 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.437494 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83762f70-bf25-4284-a707-8e304a3890fe-console-serving-cert\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:36:36.437728 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.437502 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-oauth-serving-cert\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:36:36.437728 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.437510 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83762f70-bf25-4284-a707-8e304a3890fe-trusted-ca-bundle\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:36:36.740816 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.740719 2577 generic.go:356] "Generic (PLEG): container finished" podID="83762f70-bf25-4284-a707-8e304a3890fe" containerID="c57ae4d321972db5d12611065aeb0d841a914e4bd44f603a1aac23cc1cfac75d" exitCode=0 Sep 08 00:36:36.740816 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.740775 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f8cd9b975-4wq2s" Sep 08 00:36:36.741102 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.740803 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8cd9b975-4wq2s" event={"ID":"83762f70-bf25-4284-a707-8e304a3890fe","Type":"ContainerDied","Data":"c57ae4d321972db5d12611065aeb0d841a914e4bd44f603a1aac23cc1cfac75d"} Sep 08 00:36:36.741102 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.740861 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f8cd9b975-4wq2s" event={"ID":"83762f70-bf25-4284-a707-8e304a3890fe","Type":"ContainerDied","Data":"e36a53d7c3e83a7dfe5df8145ec4d98f0044d3bb760a8b0778b7d453dea26f0e"} Sep 08 00:36:36.741102 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.740883 2577 scope.go:117] "RemoveContainer" containerID="c57ae4d321972db5d12611065aeb0d841a914e4bd44f603a1aac23cc1cfac75d" Sep 08 00:36:36.748894 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.748875 2577 scope.go:117] "RemoveContainer" containerID="c57ae4d321972db5d12611065aeb0d841a914e4bd44f603a1aac23cc1cfac75d" Sep 08 00:36:36.749186 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:36:36.749162 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c57ae4d321972db5d12611065aeb0d841a914e4bd44f603a1aac23cc1cfac75d\": container with ID starting with c57ae4d321972db5d12611065aeb0d841a914e4bd44f603a1aac23cc1cfac75d not found: ID does not exist" containerID="c57ae4d321972db5d12611065aeb0d841a914e4bd44f603a1aac23cc1cfac75d" Sep 08 00:36:36.749305 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.749188 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c57ae4d321972db5d12611065aeb0d841a914e4bd44f603a1aac23cc1cfac75d"} err="failed to get container status \"c57ae4d321972db5d12611065aeb0d841a914e4bd44f603a1aac23cc1cfac75d\": rpc error: code = NotFound desc = could not find container \"c57ae4d321972db5d12611065aeb0d841a914e4bd44f603a1aac23cc1cfac75d\": container with ID starting with c57ae4d321972db5d12611065aeb0d841a914e4bd44f603a1aac23cc1cfac75d not found: ID does not exist" Sep 08 00:36:36.761522 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.761497 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f8cd9b975-4wq2s"] Sep 08 00:36:36.764888 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.764868 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f8cd9b975-4wq2s"] Sep 08 00:36:36.789592 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:36:36.789553 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83762f70-bf25-4284-a707-8e304a3890fe" path="/var/lib/kubelet/pods/83762f70-bf25-4284-a707-8e304a3890fe/volumes" Sep 08 00:37:33.553639 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.553557 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4"] Sep 08 00:37:33.554108 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.553804 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83762f70-bf25-4284-a707-8e304a3890fe" containerName="console" Sep 08 00:37:33.554108 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.553831 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83762f70-bf25-4284-a707-8e304a3890fe" containerName="console" Sep 08 00:37:33.554108 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.553848 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71f7da9c-54f4-4ae2-b98b-bb78e3aa831e" containerName="console" Sep 08 00:37:33.554108 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.553854 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="71f7da9c-54f4-4ae2-b98b-bb78e3aa831e" containerName="console" Sep 08 00:37:33.554108 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.553898 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="71f7da9c-54f4-4ae2-b98b-bb78e3aa831e" containerName="console" Sep 08 00:37:33.554108 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.553906 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="83762f70-bf25-4284-a707-8e304a3890fe" containerName="console" Sep 08 00:37:33.556566 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.556551 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4" Sep 08 00:37:33.559028 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.558992 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 00:37:33.559266 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.559253 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-j9ctc\"" Sep 08 00:37:33.560075 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.560059 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 00:37:33.566974 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.566952 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4"] Sep 08 00:37:33.638428 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.638396 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/323fa4eb-3f25-49a4-bd12-a103b40341a0-util\") pod \"c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4\" (UID: \"323fa4eb-3f25-49a4-bd12-a103b40341a0\") " pod="openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4" Sep 08 00:37:33.638594 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.638441 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pr9l\" (UniqueName: \"kubernetes.io/projected/323fa4eb-3f25-49a4-bd12-a103b40341a0-kube-api-access-7pr9l\") pod \"c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4\" (UID: \"323fa4eb-3f25-49a4-bd12-a103b40341a0\") " pod="openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4" Sep 08 00:37:33.638594 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.638465 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/323fa4eb-3f25-49a4-bd12-a103b40341a0-bundle\") pod \"c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4\" (UID: \"323fa4eb-3f25-49a4-bd12-a103b40341a0\") " pod="openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4" Sep 08 00:37:33.738933 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.738894 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pr9l\" (UniqueName: \"kubernetes.io/projected/323fa4eb-3f25-49a4-bd12-a103b40341a0-kube-api-access-7pr9l\") pod \"c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4\" (UID: \"323fa4eb-3f25-49a4-bd12-a103b40341a0\") " pod="openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4" Sep 08 00:37:33.739153 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.738938 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/323fa4eb-3f25-49a4-bd12-a103b40341a0-bundle\") pod \"c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4\" (UID: \"323fa4eb-3f25-49a4-bd12-a103b40341a0\") " pod="openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4" Sep 08 00:37:33.739153 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.739123 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/323fa4eb-3f25-49a4-bd12-a103b40341a0-util\") pod \"c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4\" (UID: \"323fa4eb-3f25-49a4-bd12-a103b40341a0\") " pod="openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4" Sep 08 00:37:33.739391 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.739368 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/323fa4eb-3f25-49a4-bd12-a103b40341a0-bundle\") pod \"c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4\" (UID: \"323fa4eb-3f25-49a4-bd12-a103b40341a0\") " pod="openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4" Sep 08 00:37:33.739460 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.739443 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/323fa4eb-3f25-49a4-bd12-a103b40341a0-util\") pod \"c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4\" (UID: \"323fa4eb-3f25-49a4-bd12-a103b40341a0\") " pod="openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4" Sep 08 00:37:33.751182 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.751158 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pr9l\" (UniqueName: \"kubernetes.io/projected/323fa4eb-3f25-49a4-bd12-a103b40341a0-kube-api-access-7pr9l\") pod \"c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4\" (UID: \"323fa4eb-3f25-49a4-bd12-a103b40341a0\") " pod="openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4" Sep 08 00:37:33.865476 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.865389 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4" Sep 08 00:37:33.975794 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.975764 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl"] Sep 08 00:37:33.978857 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.978839 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:33.981421 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.981387 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 08 00:37:33.981717 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.981695 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 08 00:37:33.981717 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.981710 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 08 00:37:33.981840 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.981722 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 08 00:37:33.981840 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.981724 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-l4gml\"" Sep 08 00:37:33.981840 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.981699 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 08 00:37:33.982165 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.982144 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 08 00:37:33.990907 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:33.990886 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl"] Sep 08 00:37:34.139529 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.139450 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4"] Sep 08 00:37:34.142009 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.141967 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-d4qcl\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:34.142154 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.142032 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-d4qcl\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:34.142154 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.142089 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-d4qcl\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:34.142252 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.142165 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlxbk\" (UniqueName: \"kubernetes.io/projected/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-kube-api-access-zlxbk\") pod \"istiod-openshift-gateway-7b7995d5cd-d4qcl\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:34.142252 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.142216 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-d4qcl\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:34.142356 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.142259 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-d4qcl\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:34.142356 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:37:34.142322 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323fa4eb_3f25_49a4_bd12_a103b40341a0.slice/crio-b8cc9ad95008682eec49408281ed7defc606858b9eaf608fc91991f60825f807 WatchSource:0}: Error finding container b8cc9ad95008682eec49408281ed7defc606858b9eaf608fc91991f60825f807: Status 404 returned error can't find the container with id b8cc9ad95008682eec49408281ed7defc606858b9eaf608fc91991f60825f807 Sep 08 00:37:34.142356 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.142336 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-d4qcl\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:34.243148 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.243117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-d4qcl\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:34.243148 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.243151 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlxbk\" (UniqueName: \"kubernetes.io/projected/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-kube-api-access-zlxbk\") pod \"istiod-openshift-gateway-7b7995d5cd-d4qcl\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:34.243376 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.243173 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-d4qcl\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:34.243376 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.243196 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-d4qcl\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:34.243376 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.243228 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-d4qcl\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:34.243376 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.243272 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-d4qcl\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:34.243376 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.243295 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-d4qcl\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:34.243987 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.243960 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-d4qcl\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:34.245498 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.245470 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-d4qcl\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:34.245716 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.245694 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-d4qcl\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:34.245754 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.245695 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-d4qcl\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:34.245754 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.245698 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-d4qcl\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:34.252663 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.252640 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-d4qcl\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:34.254429 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.254409 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlxbk\" (UniqueName: \"kubernetes.io/projected/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-kube-api-access-zlxbk\") pod \"istiod-openshift-gateway-7b7995d5cd-d4qcl\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:34.291384 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.291362 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:34.565837 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.565813 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl"] Sep 08 00:37:34.567944 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:37:34.567922 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d2f5f4_5e97_4a59_8834_e3b2798759bb.slice/crio-45129468dfe2a7b543c9f82c78c2152c3c462699358ae537722c0a38e1348a56 WatchSource:0}: Error finding container 45129468dfe2a7b543c9f82c78c2152c3c462699358ae537722c0a38e1348a56: Status 404 returned error can't find the container with id 45129468dfe2a7b543c9f82c78c2152c3c462699358ae537722c0a38e1348a56 Sep 08 00:37:34.937359 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.937269 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4" event={"ID":"323fa4eb-3f25-49a4-bd12-a103b40341a0","Type":"ContainerStarted","Data":"b8cc9ad95008682eec49408281ed7defc606858b9eaf608fc91991f60825f807"} Sep 08 00:37:34.938689 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:34.938655 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" event={"ID":"f7d2f5f4-5e97-4a59-8834-e3b2798759bb","Type":"ContainerStarted","Data":"45129468dfe2a7b543c9f82c78c2152c3c462699358ae537722c0a38e1348a56"} Sep 08 00:37:37.266136 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:37.266088 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:37:37.266446 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:37.266180 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:37:37.954505 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:37.954463 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" event={"ID":"f7d2f5f4-5e97-4a59-8834-e3b2798759bb","Type":"ContainerStarted","Data":"f46f4f86445a0f55fbb07d8eb3cd53354779a38867ca74e28a33c60b4deae1cf"} Sep 08 00:37:37.954684 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:37.954608 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:37.975600 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:37.975547 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" podStartSLOduration=2.2796107709999998 podStartE2EDuration="4.975527882s" totalImagesPullingTime="2.695917111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:37:37.974146624 +0000 UTC m=+271.669387785" watchObservedRunningTime="2026-09-08 00:37:37.975527882 +0000 UTC m=+271.670769037" Sep 08 00:37:38.960700 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:38.960671 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:37:40.966722 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:40.966683 2577 generic.go:356] "Generic (PLEG): container finished" podID="323fa4eb-3f25-49a4-bd12-a103b40341a0" containerID="9aefcdb32c4e01e6e5ff3eb539c7074f33c33ac79c63f6894ec6a2047f60b791" exitCode=0 Sep 08 00:37:40.967142 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:40.966764 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4" event={"ID":"323fa4eb-3f25-49a4-bd12-a103b40341a0","Type":"ContainerDied","Data":"9aefcdb32c4e01e6e5ff3eb539c7074f33c33ac79c63f6894ec6a2047f60b791"} Sep 08 00:37:51.003785 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:51.003751 2577 generic.go:356] "Generic (PLEG): container finished" podID="323fa4eb-3f25-49a4-bd12-a103b40341a0" containerID="2adb12e9dd84c05c43102076b070bb18428cccc085c3e7a6665a466b0bb7ded7" exitCode=0 Sep 08 00:37:51.004194 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:51.003803 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4" event={"ID":"323fa4eb-3f25-49a4-bd12-a103b40341a0","Type":"ContainerDied","Data":"2adb12e9dd84c05c43102076b070bb18428cccc085c3e7a6665a466b0bb7ded7"} Sep 08 00:37:58.004711 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.004675 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv"] Sep 08 00:37:58.006760 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.006743 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" Sep 08 00:37:58.010067 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.010044 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 08 00:37:58.010184 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.010101 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Sep 08 00:37:58.010502 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.010484 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Sep 08 00:37:58.010925 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.010907 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 08 00:37:58.011028 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.010914 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-4zkls\"" Sep 08 00:37:58.011028 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.010960 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Sep 08 00:37:58.029243 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.029218 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e8a588fa-2622-4fde-bdab-d7f5a7474767-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-677bv\" (UID: \"e8a588fa-2622-4fde-bdab-d7f5a7474767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" Sep 08 00:37:58.029336 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.029248 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e8a588fa-2622-4fde-bdab-d7f5a7474767-manifests\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-677bv\" (UID: \"e8a588fa-2622-4fde-bdab-d7f5a7474767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" Sep 08 00:37:58.029336 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.029279 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8a588fa-2622-4fde-bdab-d7f5a7474767-tmp\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-677bv\" (UID: \"e8a588fa-2622-4fde-bdab-d7f5a7474767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" Sep 08 00:37:58.029402 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.029346 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e8a588fa-2622-4fde-bdab-d7f5a7474767-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-677bv\" (UID: \"e8a588fa-2622-4fde-bdab-d7f5a7474767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" Sep 08 00:37:58.029402 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.029392 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8a588fa-2622-4fde-bdab-d7f5a7474767-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-677bv\" (UID: \"e8a588fa-2622-4fde-bdab-d7f5a7474767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" Sep 08 00:37:58.029467 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.029425 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh764\" (UniqueName: \"kubernetes.io/projected/e8a588fa-2622-4fde-bdab-d7f5a7474767-kube-api-access-gh764\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-677bv\" (UID: \"e8a588fa-2622-4fde-bdab-d7f5a7474767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" Sep 08 00:37:58.032689 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.032665 2577 generic.go:356] "Generic (PLEG): container finished" podID="323fa4eb-3f25-49a4-bd12-a103b40341a0" containerID="a1711c8f237c98a4f8b168eaaee47e06162a5efcf2fe0f1120a52f8915a23389" exitCode=0 Sep 08 00:37:58.032781 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.032727 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4" event={"ID":"323fa4eb-3f25-49a4-bd12-a103b40341a0","Type":"ContainerDied","Data":"a1711c8f237c98a4f8b168eaaee47e06162a5efcf2fe0f1120a52f8915a23389"} Sep 08 00:37:58.033654 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.033638 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv"] Sep 08 00:37:58.130288 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.130252 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8a588fa-2622-4fde-bdab-d7f5a7474767-tmp\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-677bv\" (UID: \"e8a588fa-2622-4fde-bdab-d7f5a7474767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" Sep 08 00:37:58.130288 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.130286 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e8a588fa-2622-4fde-bdab-d7f5a7474767-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-677bv\" (UID: \"e8a588fa-2622-4fde-bdab-d7f5a7474767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" Sep 08 00:37:58.130519 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.130304 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8a588fa-2622-4fde-bdab-d7f5a7474767-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-677bv\" (UID: \"e8a588fa-2622-4fde-bdab-d7f5a7474767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" Sep 08 00:37:58.130519 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.130323 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh764\" (UniqueName: \"kubernetes.io/projected/e8a588fa-2622-4fde-bdab-d7f5a7474767-kube-api-access-gh764\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-677bv\" (UID: \"e8a588fa-2622-4fde-bdab-d7f5a7474767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" Sep 08 00:37:58.130624 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.130554 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e8a588fa-2622-4fde-bdab-d7f5a7474767-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-677bv\" (UID: \"e8a588fa-2622-4fde-bdab-d7f5a7474767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" Sep 08 00:37:58.130624 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.130605 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e8a588fa-2622-4fde-bdab-d7f5a7474767-manifests\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-677bv\" (UID: \"e8a588fa-2622-4fde-bdab-d7f5a7474767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" Sep 08 00:37:58.130926 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.130889 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e8a588fa-2622-4fde-bdab-d7f5a7474767-manifests\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-677bv\" (UID: \"e8a588fa-2622-4fde-bdab-d7f5a7474767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" Sep 08 00:37:58.132778 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.132756 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8a588fa-2622-4fde-bdab-d7f5a7474767-tmp\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-677bv\" (UID: \"e8a588fa-2622-4fde-bdab-d7f5a7474767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" Sep 08 00:37:58.133042 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.133023 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e8a588fa-2622-4fde-bdab-d7f5a7474767-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-677bv\" (UID: \"e8a588fa-2622-4fde-bdab-d7f5a7474767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" Sep 08 00:37:58.133188 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.133133 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e8a588fa-2622-4fde-bdab-d7f5a7474767-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-677bv\" (UID: \"e8a588fa-2622-4fde-bdab-d7f5a7474767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" Sep 08 00:37:58.133252 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.133226 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8a588fa-2622-4fde-bdab-d7f5a7474767-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-677bv\" (UID: \"e8a588fa-2622-4fde-bdab-d7f5a7474767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" Sep 08 00:37:58.141281 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.141260 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh764\" (UniqueName: \"kubernetes.io/projected/e8a588fa-2622-4fde-bdab-d7f5a7474767-kube-api-access-gh764\") pod \"opendatahub-operator-controller-manager-7c7d4c4566-677bv\" (UID: \"e8a588fa-2622-4fde-bdab-d7f5a7474767\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" Sep 08 00:37:58.316318 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.316292 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" Sep 08 00:37:58.633487 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:58.633349 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv"] Sep 08 00:37:58.640476 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:37:58.640435 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a588fa_2622_4fde_bdab_d7f5a7474767.slice/crio-edb256929f3aa143b5ef354b24d7cbec21defb9e9da83f13e9acd2b208b17afc WatchSource:0}: Error finding container edb256929f3aa143b5ef354b24d7cbec21defb9e9da83f13e9acd2b208b17afc: Status 404 returned error can't find the container with id edb256929f3aa143b5ef354b24d7cbec21defb9e9da83f13e9acd2b208b17afc Sep 08 00:37:59.037645 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:59.037606 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" event={"ID":"e8a588fa-2622-4fde-bdab-d7f5a7474767","Type":"ContainerStarted","Data":"edb256929f3aa143b5ef354b24d7cbec21defb9e9da83f13e9acd2b208b17afc"} Sep 08 00:37:59.154585 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:59.154535 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4" Sep 08 00:37:59.238026 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:59.237971 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/323fa4eb-3f25-49a4-bd12-a103b40341a0-bundle\") pod \"323fa4eb-3f25-49a4-bd12-a103b40341a0\" (UID: \"323fa4eb-3f25-49a4-bd12-a103b40341a0\") " Sep 08 00:37:59.238207 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:59.238050 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/323fa4eb-3f25-49a4-bd12-a103b40341a0-util\") pod \"323fa4eb-3f25-49a4-bd12-a103b40341a0\" (UID: \"323fa4eb-3f25-49a4-bd12-a103b40341a0\") " Sep 08 00:37:59.238207 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:59.238124 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pr9l\" (UniqueName: \"kubernetes.io/projected/323fa4eb-3f25-49a4-bd12-a103b40341a0-kube-api-access-7pr9l\") pod \"323fa4eb-3f25-49a4-bd12-a103b40341a0\" (UID: \"323fa4eb-3f25-49a4-bd12-a103b40341a0\") " Sep 08 00:37:59.238388 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:59.238360 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323fa4eb-3f25-49a4-bd12-a103b40341a0-bundle" (OuterVolumeSpecName: "bundle") pod "323fa4eb-3f25-49a4-bd12-a103b40341a0" (UID: "323fa4eb-3f25-49a4-bd12-a103b40341a0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:37:59.240649 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:59.240618 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323fa4eb-3f25-49a4-bd12-a103b40341a0-kube-api-access-7pr9l" (OuterVolumeSpecName: "kube-api-access-7pr9l") pod "323fa4eb-3f25-49a4-bd12-a103b40341a0" (UID: "323fa4eb-3f25-49a4-bd12-a103b40341a0"). InnerVolumeSpecName "kube-api-access-7pr9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:37:59.244144 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:59.244110 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/323fa4eb-3f25-49a4-bd12-a103b40341a0-util" (OuterVolumeSpecName: "util") pod "323fa4eb-3f25-49a4-bd12-a103b40341a0" (UID: "323fa4eb-3f25-49a4-bd12-a103b40341a0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:37:59.339380 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:59.339302 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pr9l\" (UniqueName: \"kubernetes.io/projected/323fa4eb-3f25-49a4-bd12-a103b40341a0-kube-api-access-7pr9l\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:37:59.339380 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:59.339336 2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/323fa4eb-3f25-49a4-bd12-a103b40341a0-bundle\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:37:59.339380 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:37:59.339349 2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/323fa4eb-3f25-49a4-bd12-a103b40341a0-util\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:38:00.044264 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:00.044229 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4" event={"ID":"323fa4eb-3f25-49a4-bd12-a103b40341a0","Type":"ContainerDied","Data":"b8cc9ad95008682eec49408281ed7defc606858b9eaf608fc91991f60825f807"} Sep 08 00:38:00.044264 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:00.044260 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c2ca89134faa49158137edb0141b62ea0c6a854657aff316cf72d9c78ep2vs4" Sep 08 00:38:00.044804 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:00.044276 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8cc9ad95008682eec49408281ed7defc606858b9eaf608fc91991f60825f807" Sep 08 00:38:02.053657 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:02.053624 2577 generic.go:356] "Generic (PLEG): container finished" podID="e8a588fa-2622-4fde-bdab-d7f5a7474767" containerID="4191c8d983a3458cfe91b584dc076778c866fbdfb874dd3b067ac29da10fb920" exitCode=0 Sep 08 00:38:02.054051 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:02.053703 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" event={"ID":"e8a588fa-2622-4fde-bdab-d7f5a7474767","Type":"ContainerDied","Data":"4191c8d983a3458cfe91b584dc076778c866fbdfb874dd3b067ac29da10fb920"} Sep 08 00:38:03.059202 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:03.059164 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" event={"ID":"e8a588fa-2622-4fde-bdab-d7f5a7474767","Type":"ContainerStarted","Data":"df705cec636eb0492801cf426c4871e6dac25217e55fd89e941063823e74b31b"} Sep 08 00:38:03.059665 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:03.059289 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" Sep 08 00:38:03.085883 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:03.085840 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" podStartSLOduration=2.456965398 podStartE2EDuration="6.085824674s" totalImagesPullingTime="3.150234191s" totalInitContainerRuntime="478.625085ms" isStatefulPod=false podCreationTimestamp="2026-09-08 00:37:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:38:03.083979871 +0000 UTC m=+296.779221051" watchObservedRunningTime="2026-09-08 00:38:03.085824674 +0000 UTC m=+296.781065834" Sep 08 00:38:06.664070 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:06.664041 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9nb9x_a2e160ae-d75d-4959-a5ab-d238b4113736/console-operator/1.log" Sep 08 00:38:06.664545 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:06.664309 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9nb9x_a2e160ae-d75d-4959-a5ab-d238b4113736/console-operator/1.log" Sep 08 00:38:06.674615 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:06.674596 2577 kubelet.go:1648] "Image garbage collection succeeded" Sep 08 00:38:14.064166 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:14.064134 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c7d4c4566-677bv" Sep 08 00:38:28.604887 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:28.604850 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-58c5b795c4-npd8z"] Sep 08 00:38:28.605479 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:28.605194 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="323fa4eb-3f25-49a4-bd12-a103b40341a0" containerName="extract" Sep 08 00:38:28.605479 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:28.605213 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="323fa4eb-3f25-49a4-bd12-a103b40341a0" containerName="extract" Sep 08 00:38:28.605479 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:28.605230 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="323fa4eb-3f25-49a4-bd12-a103b40341a0" containerName="pull" Sep 08 00:38:28.605479 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:28.605238 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="323fa4eb-3f25-49a4-bd12-a103b40341a0" containerName="pull" Sep 08 00:38:28.605479 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:28.605253 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="323fa4eb-3f25-49a4-bd12-a103b40341a0" containerName="util" Sep 08 00:38:28.605479 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:28.605262 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="323fa4eb-3f25-49a4-bd12-a103b40341a0" containerName="util" Sep 08 00:38:28.605479 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:28.605341 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="323fa4eb-3f25-49a4-bd12-a103b40341a0" containerName="extract" Sep 08 00:38:28.616583 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:28.616557 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58c5b795c4-npd8z" Sep 08 00:38:28.619238 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:28.619209 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 08 00:38:28.619673 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:28.619651 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-q8kwd\"" Sep 08 00:38:28.621191 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:28.621169 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 08 00:38:28.622676 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:28.622655 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58c5b795c4-npd8z"] Sep 08 00:38:28.666399 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:28.666370 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpwvg\" (UniqueName: \"kubernetes.io/projected/fbf31006-fc5d-4957-9766-b89b65cee90a-kube-api-access-vpwvg\") pod \"kube-auth-proxy-58c5b795c4-npd8z\" (UID: \"fbf31006-fc5d-4957-9766-b89b65cee90a\") " pod="openshift-ingress/kube-auth-proxy-58c5b795c4-npd8z" Sep 08 00:38:28.666548 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:28.666411 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf31006-fc5d-4957-9766-b89b65cee90a-tls-certs\") pod \"kube-auth-proxy-58c5b795c4-npd8z\" (UID: \"fbf31006-fc5d-4957-9766-b89b65cee90a\") " pod="openshift-ingress/kube-auth-proxy-58c5b795c4-npd8z" Sep 08 00:38:28.666548 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:28.666534 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbf31006-fc5d-4957-9766-b89b65cee90a-tmp\") pod \"kube-auth-proxy-58c5b795c4-npd8z\" (UID: \"fbf31006-fc5d-4957-9766-b89b65cee90a\") " pod="openshift-ingress/kube-auth-proxy-58c5b795c4-npd8z" Sep 08 00:38:28.767814 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:28.767774 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbf31006-fc5d-4957-9766-b89b65cee90a-tmp\") pod \"kube-auth-proxy-58c5b795c4-npd8z\" (UID: \"fbf31006-fc5d-4957-9766-b89b65cee90a\") " pod="openshift-ingress/kube-auth-proxy-58c5b795c4-npd8z" Sep 08 00:38:28.768010 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:28.767831 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpwvg\" (UniqueName: \"kubernetes.io/projected/fbf31006-fc5d-4957-9766-b89b65cee90a-kube-api-access-vpwvg\") pod \"kube-auth-proxy-58c5b795c4-npd8z\" (UID: \"fbf31006-fc5d-4957-9766-b89b65cee90a\") " pod="openshift-ingress/kube-auth-proxy-58c5b795c4-npd8z" Sep 08 00:38:28.768010 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:28.767865 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf31006-fc5d-4957-9766-b89b65cee90a-tls-certs\") pod \"kube-auth-proxy-58c5b795c4-npd8z\" (UID: \"fbf31006-fc5d-4957-9766-b89b65cee90a\") " pod="openshift-ingress/kube-auth-proxy-58c5b795c4-npd8z" Sep 08 00:38:28.768166 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:38:28.768025 2577 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 08 00:38:28.768166 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:38:28.768107 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbf31006-fc5d-4957-9766-b89b65cee90a-tls-certs podName:fbf31006-fc5d-4957-9766-b89b65cee90a nodeName:}" failed. No retries permitted until 2026-09-08 00:38:29.268085104 +0000 UTC m=+322.963326246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fbf31006-fc5d-4957-9766-b89b65cee90a-tls-certs") pod "kube-auth-proxy-58c5b795c4-npd8z" (UID: "fbf31006-fc5d-4957-9766-b89b65cee90a") : secret "kube-auth-proxy-tls" not found Sep 08 00:38:28.769901 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:28.769872 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbf31006-fc5d-4957-9766-b89b65cee90a-tmp\") pod \"kube-auth-proxy-58c5b795c4-npd8z\" (UID: \"fbf31006-fc5d-4957-9766-b89b65cee90a\") " pod="openshift-ingress/kube-auth-proxy-58c5b795c4-npd8z" Sep 08 00:38:28.779355 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:28.779331 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpwvg\" (UniqueName: \"kubernetes.io/projected/fbf31006-fc5d-4957-9766-b89b65cee90a-kube-api-access-vpwvg\") pod \"kube-auth-proxy-58c5b795c4-npd8z\" (UID: \"fbf31006-fc5d-4957-9766-b89b65cee90a\") " pod="openshift-ingress/kube-auth-proxy-58c5b795c4-npd8z" Sep 08 00:38:29.271416 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:29.271381 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf31006-fc5d-4957-9766-b89b65cee90a-tls-certs\") pod \"kube-auth-proxy-58c5b795c4-npd8z\" (UID: \"fbf31006-fc5d-4957-9766-b89b65cee90a\") " pod="openshift-ingress/kube-auth-proxy-58c5b795c4-npd8z" Sep 08 00:38:29.273702 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:29.273680 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbf31006-fc5d-4957-9766-b89b65cee90a-tls-certs\") pod \"kube-auth-proxy-58c5b795c4-npd8z\" (UID: \"fbf31006-fc5d-4957-9766-b89b65cee90a\") " pod="openshift-ingress/kube-auth-proxy-58c5b795c4-npd8z" Sep 08 00:38:29.528836 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:29.528810 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58c5b795c4-npd8z" Sep 08 00:38:29.814105 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:29.814022 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58c5b795c4-npd8z"] Sep 08 00:38:29.817375 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:38:29.817342 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf31006_fc5d_4957_9766_b89b65cee90a.slice/crio-c6dad29671977c684499aa3f494c9ac38c805882e9a48ec91f353760330ec63d WatchSource:0}: Error finding container c6dad29671977c684499aa3f494c9ac38c805882e9a48ec91f353760330ec63d: Status 404 returned error can't find the container with id c6dad29671977c684499aa3f494c9ac38c805882e9a48ec91f353760330ec63d Sep 08 00:38:29.819261 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:29.819243 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 00:38:30.156184 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:30.156097 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58c5b795c4-npd8z" event={"ID":"fbf31006-fc5d-4957-9766-b89b65cee90a","Type":"ContainerStarted","Data":"c6dad29671977c684499aa3f494c9ac38c805882e9a48ec91f353760330ec63d"} Sep 08 00:38:33.169514 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:33.169475 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58c5b795c4-npd8z" event={"ID":"fbf31006-fc5d-4957-9766-b89b65cee90a","Type":"ContainerStarted","Data":"59f0a7022251a9196f76edfe0d359b5ee272cae96dc8501287d6bd8c79b64356"} Sep 08 00:38:33.186851 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:33.186802 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-58c5b795c4-npd8z" podStartSLOduration=2.710419862 podStartE2EDuration="5.186786813s" totalImagesPullingTime="2.476366951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:38:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:38:33.186133174 +0000 UTC m=+326.881374330" watchObservedRunningTime="2026-09-08 00:38:33.186786813 +0000 UTC m=+326.882027972" Sep 08 00:38:42.599499 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:42.599465 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bfjjj"] Sep 08 00:38:42.603683 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:42.603667 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bfjjj" Sep 08 00:38:42.606075 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:42.606054 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 08 00:38:42.606186 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:42.606107 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 08 00:38:42.607166 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:42.607144 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-lcc5v\"" Sep 08 00:38:42.612086 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:42.612065 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bfjjj"] Sep 08 00:38:42.682344 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:42.682312 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98ee6bc1-4623-4915-9db8-3e4869a9ab5b-bound-sa-token\") pod \"cert-manager-55b4d954d5-bfjjj\" (UID: \"98ee6bc1-4623-4915-9db8-3e4869a9ab5b\") " pod="cert-manager/cert-manager-55b4d954d5-bfjjj" Sep 08 00:38:42.682497 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:42.682375 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hntts\" (UniqueName: \"kubernetes.io/projected/98ee6bc1-4623-4915-9db8-3e4869a9ab5b-kube-api-access-hntts\") pod \"cert-manager-55b4d954d5-bfjjj\" (UID: \"98ee6bc1-4623-4915-9db8-3e4869a9ab5b\") " pod="cert-manager/cert-manager-55b4d954d5-bfjjj" Sep 08 00:38:42.783625 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:42.783572 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98ee6bc1-4623-4915-9db8-3e4869a9ab5b-bound-sa-token\") pod \"cert-manager-55b4d954d5-bfjjj\" (UID: \"98ee6bc1-4623-4915-9db8-3e4869a9ab5b\") " pod="cert-manager/cert-manager-55b4d954d5-bfjjj" Sep 08 00:38:42.783797 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:42.783642 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hntts\" (UniqueName: \"kubernetes.io/projected/98ee6bc1-4623-4915-9db8-3e4869a9ab5b-kube-api-access-hntts\") pod \"cert-manager-55b4d954d5-bfjjj\" (UID: \"98ee6bc1-4623-4915-9db8-3e4869a9ab5b\") " pod="cert-manager/cert-manager-55b4d954d5-bfjjj" Sep 08 00:38:42.791325 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:42.791297 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/98ee6bc1-4623-4915-9db8-3e4869a9ab5b-bound-sa-token\") pod \"cert-manager-55b4d954d5-bfjjj\" (UID: \"98ee6bc1-4623-4915-9db8-3e4869a9ab5b\") " pod="cert-manager/cert-manager-55b4d954d5-bfjjj" Sep 08 00:38:42.791464 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:42.791445 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hntts\" (UniqueName: \"kubernetes.io/projected/98ee6bc1-4623-4915-9db8-3e4869a9ab5b-kube-api-access-hntts\") pod \"cert-manager-55b4d954d5-bfjjj\" (UID: \"98ee6bc1-4623-4915-9db8-3e4869a9ab5b\") " pod="cert-manager/cert-manager-55b4d954d5-bfjjj" Sep 08 00:38:42.913513 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:42.913423 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-bfjjj" Sep 08 00:38:43.188911 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:43.188877 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-bfjjj"] Sep 08 00:38:43.190044 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:38:43.190011 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98ee6bc1_4623_4915_9db8_3e4869a9ab5b.slice/crio-a1ce27eb8bf53baad3db86e6dc3ccf0bd2cea2591da5088f8e0bd6de9fdbc421 WatchSource:0}: Error finding container a1ce27eb8bf53baad3db86e6dc3ccf0bd2cea2591da5088f8e0bd6de9fdbc421: Status 404 returned error can't find the container with id a1ce27eb8bf53baad3db86e6dc3ccf0bd2cea2591da5088f8e0bd6de9fdbc421 Sep 08 00:38:43.213655 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:43.213631 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bfjjj" event={"ID":"98ee6bc1-4623-4915-9db8-3e4869a9ab5b","Type":"ContainerStarted","Data":"a1ce27eb8bf53baad3db86e6dc3ccf0bd2cea2591da5088f8e0bd6de9fdbc421"} Sep 08 00:38:47.230118 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:47.230082 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-bfjjj" event={"ID":"98ee6bc1-4623-4915-9db8-3e4869a9ab5b","Type":"ContainerStarted","Data":"38a6f43fc850fe3858fbdd643467c2cf3bca2fdb68b8adab230a90442fa4e191"} Sep 08 00:38:47.249960 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:38:47.249914 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-bfjjj" podStartSLOduration=2.240439199 podStartE2EDuration="5.249898939s" totalImagesPullingTime="3.00945974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:38:47.248560145 +0000 UTC m=+340.943801305" watchObservedRunningTime="2026-09-08 00:38:47.249898939 +0000 UTC m=+340.945140098" Sep 08 00:39:13.563124 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:13.563087 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-v8gn6"] Sep 08 00:39:13.566989 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:13.566967 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-v8gn6" Sep 08 00:39:13.569776 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:13.569752 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 08 00:39:13.569934 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:13.569919 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kwxr7\"" Sep 08 00:39:13.571059 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:13.571042 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 08 00:39:13.577010 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:13.576977 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-v8gn6"] Sep 08 00:39:13.606507 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:13.606471 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46bsj\" (UniqueName: \"kubernetes.io/projected/62e66c7e-a81d-4b97-badf-2d4f5d3c821f-kube-api-access-46bsj\") pod \"authorino-b54df94b4-v8gn6\" (UID: \"62e66c7e-a81d-4b97-badf-2d4f5d3c821f\") " pod="kuadrant-system/authorino-b54df94b4-v8gn6" Sep 08 00:39:13.707746 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:13.707714 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46bsj\" (UniqueName: \"kubernetes.io/projected/62e66c7e-a81d-4b97-badf-2d4f5d3c821f-kube-api-access-46bsj\") pod \"authorino-b54df94b4-v8gn6\" (UID: \"62e66c7e-a81d-4b97-badf-2d4f5d3c821f\") " pod="kuadrant-system/authorino-b54df94b4-v8gn6" Sep 08 00:39:13.716011 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:13.715967 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46bsj\" (UniqueName: \"kubernetes.io/projected/62e66c7e-a81d-4b97-badf-2d4f5d3c821f-kube-api-access-46bsj\") pod \"authorino-b54df94b4-v8gn6\" (UID: \"62e66c7e-a81d-4b97-badf-2d4f5d3c821f\") " pod="kuadrant-system/authorino-b54df94b4-v8gn6" Sep 08 00:39:13.876619 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:13.876535 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-v8gn6" Sep 08 00:39:14.145915 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:14.145889 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-v8gn6"] Sep 08 00:39:14.148737 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:39:14.148709 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62e66c7e_a81d_4b97_badf_2d4f5d3c821f.slice/crio-3d4fb4ab06ae061413258ad73e7de6b8219902a6fda91718cebec3aa5947b77a WatchSource:0}: Error finding container 3d4fb4ab06ae061413258ad73e7de6b8219902a6fda91718cebec3aa5947b77a: Status 404 returned error can't find the container with id 3d4fb4ab06ae061413258ad73e7de6b8219902a6fda91718cebec3aa5947b77a Sep 08 00:39:14.337177 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:14.337144 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-v8gn6" event={"ID":"62e66c7e-a81d-4b97-badf-2d4f5d3c821f","Type":"ContainerStarted","Data":"3d4fb4ab06ae061413258ad73e7de6b8219902a6fda91718cebec3aa5947b77a"} Sep 08 00:39:17.143544 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:17.143509 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-v8gn6"] Sep 08 00:39:18.361675 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:18.361634 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-v8gn6" event={"ID":"62e66c7e-a81d-4b97-badf-2d4f5d3c821f","Type":"ContainerStarted","Data":"8acb513d38c58be937c092fdec3078778889185e4a51449ff1de0f8c46603753"} Sep 08 00:39:18.362140 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:18.361695 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-v8gn6" podUID="62e66c7e-a81d-4b97-badf-2d4f5d3c821f" containerName="authorino" containerID="cri-o://8acb513d38c58be937c092fdec3078778889185e4a51449ff1de0f8c46603753" gracePeriod=30 Sep 08 00:39:18.590651 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:18.590625 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-v8gn6" Sep 08 00:39:18.656754 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:18.656673 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46bsj\" (UniqueName: \"kubernetes.io/projected/62e66c7e-a81d-4b97-badf-2d4f5d3c821f-kube-api-access-46bsj\") pod \"62e66c7e-a81d-4b97-badf-2d4f5d3c821f\" (UID: \"62e66c7e-a81d-4b97-badf-2d4f5d3c821f\") " Sep 08 00:39:18.658694 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:18.658668 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e66c7e-a81d-4b97-badf-2d4f5d3c821f-kube-api-access-46bsj" (OuterVolumeSpecName: "kube-api-access-46bsj") pod "62e66c7e-a81d-4b97-badf-2d4f5d3c821f" (UID: "62e66c7e-a81d-4b97-badf-2d4f5d3c821f"). InnerVolumeSpecName "kube-api-access-46bsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:39:18.757285 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:18.757248 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46bsj\" (UniqueName: \"kubernetes.io/projected/62e66c7e-a81d-4b97-badf-2d4f5d3c821f-kube-api-access-46bsj\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:39:19.367089 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.367059 2577 generic.go:356] "Generic (PLEG): container finished" podID="62e66c7e-a81d-4b97-badf-2d4f5d3c821f" containerID="8acb513d38c58be937c092fdec3078778889185e4a51449ff1de0f8c46603753" exitCode=0 Sep 08 00:39:19.367492 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.367101 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-v8gn6" Sep 08 00:39:19.367492 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.367112 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-v8gn6" event={"ID":"62e66c7e-a81d-4b97-badf-2d4f5d3c821f","Type":"ContainerDied","Data":"8acb513d38c58be937c092fdec3078778889185e4a51449ff1de0f8c46603753"} Sep 08 00:39:19.367492 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.367144 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-v8gn6" event={"ID":"62e66c7e-a81d-4b97-badf-2d4f5d3c821f","Type":"ContainerDied","Data":"3d4fb4ab06ae061413258ad73e7de6b8219902a6fda91718cebec3aa5947b77a"} Sep 08 00:39:19.367492 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.367163 2577 scope.go:117] "RemoveContainer" containerID="8acb513d38c58be937c092fdec3078778889185e4a51449ff1de0f8c46603753" Sep 08 00:39:19.375375 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.375356 2577 scope.go:117] "RemoveContainer" containerID="8acb513d38c58be937c092fdec3078778889185e4a51449ff1de0f8c46603753" Sep 08 00:39:19.375648 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:39:19.375628 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8acb513d38c58be937c092fdec3078778889185e4a51449ff1de0f8c46603753\": container with ID starting with 8acb513d38c58be937c092fdec3078778889185e4a51449ff1de0f8c46603753 not found: ID does not exist" containerID="8acb513d38c58be937c092fdec3078778889185e4a51449ff1de0f8c46603753" Sep 08 00:39:19.375692 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.375657 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8acb513d38c58be937c092fdec3078778889185e4a51449ff1de0f8c46603753"} err="failed to get container status \"8acb513d38c58be937c092fdec3078778889185e4a51449ff1de0f8c46603753\": rpc error: code = NotFound desc = could not find container \"8acb513d38c58be937c092fdec3078778889185e4a51449ff1de0f8c46603753\": container with ID starting with 8acb513d38c58be937c092fdec3078778889185e4a51449ff1de0f8c46603753 not found: ID does not exist" Sep 08 00:39:19.426434 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.426402 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-v8gn6"] Sep 08 00:39:19.440218 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.440185 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-v8gn6"] Sep 08 00:39:19.523327 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.523290 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-55885fcb99-2fcml"] Sep 08 00:39:19.523614 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.523600 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62e66c7e-a81d-4b97-badf-2d4f5d3c821f" containerName="authorino" Sep 08 00:39:19.523658 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.523616 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="62e66c7e-a81d-4b97-badf-2d4f5d3c821f" containerName="authorino" Sep 08 00:39:19.523691 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.523663 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="62e66c7e-a81d-4b97-badf-2d4f5d3c821f" containerName="authorino" Sep 08 00:39:19.527927 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.527909 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-55885fcb99-2fcml" Sep 08 00:39:19.534434 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.534393 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Sep 08 00:39:19.535059 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.534894 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Sep 08 00:39:19.535059 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.534905 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Sep 08 00:39:19.535059 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.534937 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Sep 08 00:39:19.535243 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.535196 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Sep 08 00:39:19.535616 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.535597 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-24w4b\"" Sep 08 00:39:19.555013 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.554983 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-55885fcb99-2fcml"] Sep 08 00:39:19.563882 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.563860 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b732ebd-9724-46b7-b315-ae24c46ef456-metrics-certs\") pod \"jobset-controller-manager-55885fcb99-2fcml\" (UID: \"4b732ebd-9724-46b7-b315-ae24c46ef456\") " pod="openshift-jobset-operator/jobset-controller-manager-55885fcb99-2fcml" Sep 08 00:39:19.564005 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.563915 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znks7\" (UniqueName: \"kubernetes.io/projected/4b732ebd-9724-46b7-b315-ae24c46ef456-kube-api-access-znks7\") pod \"jobset-controller-manager-55885fcb99-2fcml\" (UID: \"4b732ebd-9724-46b7-b315-ae24c46ef456\") " pod="openshift-jobset-operator/jobset-controller-manager-55885fcb99-2fcml" Sep 08 00:39:19.564005 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.563976 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4b732ebd-9724-46b7-b315-ae24c46ef456-manager-config\") pod \"jobset-controller-manager-55885fcb99-2fcml\" (UID: \"4b732ebd-9724-46b7-b315-ae24c46ef456\") " pod="openshift-jobset-operator/jobset-controller-manager-55885fcb99-2fcml" Sep 08 00:39:19.564291 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.564021 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b732ebd-9724-46b7-b315-ae24c46ef456-cert\") pod \"jobset-controller-manager-55885fcb99-2fcml\" (UID: \"4b732ebd-9724-46b7-b315-ae24c46ef456\") " pod="openshift-jobset-operator/jobset-controller-manager-55885fcb99-2fcml" Sep 08 00:39:19.665013 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.664892 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4b732ebd-9724-46b7-b315-ae24c46ef456-manager-config\") pod \"jobset-controller-manager-55885fcb99-2fcml\" (UID: \"4b732ebd-9724-46b7-b315-ae24c46ef456\") " pod="openshift-jobset-operator/jobset-controller-manager-55885fcb99-2fcml" Sep 08 00:39:19.665013 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.664936 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b732ebd-9724-46b7-b315-ae24c46ef456-cert\") pod \"jobset-controller-manager-55885fcb99-2fcml\" (UID: \"4b732ebd-9724-46b7-b315-ae24c46ef456\") " pod="openshift-jobset-operator/jobset-controller-manager-55885fcb99-2fcml" Sep 08 00:39:19.665013 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.664964 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b732ebd-9724-46b7-b315-ae24c46ef456-metrics-certs\") pod \"jobset-controller-manager-55885fcb99-2fcml\" (UID: \"4b732ebd-9724-46b7-b315-ae24c46ef456\") " pod="openshift-jobset-operator/jobset-controller-manager-55885fcb99-2fcml" Sep 08 00:39:19.665282 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.665117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znks7\" (UniqueName: \"kubernetes.io/projected/4b732ebd-9724-46b7-b315-ae24c46ef456-kube-api-access-znks7\") pod \"jobset-controller-manager-55885fcb99-2fcml\" (UID: \"4b732ebd-9724-46b7-b315-ae24c46ef456\") " pod="openshift-jobset-operator/jobset-controller-manager-55885fcb99-2fcml" Sep 08 00:39:19.665955 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.665925 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4b732ebd-9724-46b7-b315-ae24c46ef456-manager-config\") pod \"jobset-controller-manager-55885fcb99-2fcml\" (UID: \"4b732ebd-9724-46b7-b315-ae24c46ef456\") " pod="openshift-jobset-operator/jobset-controller-manager-55885fcb99-2fcml" Sep 08 00:39:19.667744 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.667722 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b732ebd-9724-46b7-b315-ae24c46ef456-metrics-certs\") pod \"jobset-controller-manager-55885fcb99-2fcml\" (UID: \"4b732ebd-9724-46b7-b315-ae24c46ef456\") " pod="openshift-jobset-operator/jobset-controller-manager-55885fcb99-2fcml" Sep 08 00:39:19.667883 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.667862 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b732ebd-9724-46b7-b315-ae24c46ef456-cert\") pod \"jobset-controller-manager-55885fcb99-2fcml\" (UID: \"4b732ebd-9724-46b7-b315-ae24c46ef456\") " pod="openshift-jobset-operator/jobset-controller-manager-55885fcb99-2fcml" Sep 08 00:39:19.688096 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.688026 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znks7\" (UniqueName: \"kubernetes.io/projected/4b732ebd-9724-46b7-b315-ae24c46ef456-kube-api-access-znks7\") pod \"jobset-controller-manager-55885fcb99-2fcml\" (UID: \"4b732ebd-9724-46b7-b315-ae24c46ef456\") " pod="openshift-jobset-operator/jobset-controller-manager-55885fcb99-2fcml" Sep 08 00:39:19.838136 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:19.838103 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-55885fcb99-2fcml" Sep 08 00:39:20.118950 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:20.118928 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-55885fcb99-2fcml"] Sep 08 00:39:20.120450 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:39:20.120424 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b732ebd_9724_46b7_b315_ae24c46ef456.slice/crio-67c51fe35670284192c533c54f3d2be4fd9ac83f09e4d3db0059deb214878d04 WatchSource:0}: Error finding container 67c51fe35670284192c533c54f3d2be4fd9ac83f09e4d3db0059deb214878d04: Status 404 returned error can't find the container with id 67c51fe35670284192c533c54f3d2be4fd9ac83f09e4d3db0059deb214878d04 Sep 08 00:39:20.372484 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:20.372394 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-55885fcb99-2fcml" event={"ID":"4b732ebd-9724-46b7-b315-ae24c46ef456","Type":"ContainerStarted","Data":"67c51fe35670284192c533c54f3d2be4fd9ac83f09e4d3db0059deb214878d04"} Sep 08 00:39:20.789578 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:20.789546 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62e66c7e-a81d-4b97-badf-2d4f5d3c821f" path="/var/lib/kubelet/pods/62e66c7e-a81d-4b97-badf-2d4f5d3c821f/volumes" Sep 08 00:39:23.387932 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:23.387853 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-55885fcb99-2fcml" event={"ID":"4b732ebd-9724-46b7-b315-ae24c46ef456","Type":"ContainerStarted","Data":"c9aae5d5333cfc9838454f2309651774a5502124c160586e47d27a60d543556e"} Sep 08 00:39:23.388514 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:23.387984 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-55885fcb99-2fcml" Sep 08 00:39:23.438571 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:23.438480 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-55885fcb99-2fcml" podStartSLOduration=1.3955145820000001 podStartE2EDuration="4.438461187s" totalImagesPullingTime="3.042946605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:39:23.425677316 +0000 UTC m=+377.120918475" watchObservedRunningTime="2026-09-08 00:39:23.438461187 +0000 UTC m=+377.133702346" Sep 08 00:39:24.545245 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.545210 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx"] Sep 08 00:39:24.548532 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.548510 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:24.564742 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.564712 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx"] Sep 08 00:39:24.609017 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.608960 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7zzz\" (UniqueName: \"kubernetes.io/projected/14d4c77a-4999-4d40-803a-80b4f77aad5d-kube-api-access-m7zzz\") pod \"istiod-openshift-gateway-5677ccb49b-h2gwx\" (UID: \"14d4c77a-4999-4d40-803a-80b4f77aad5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:24.609314 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.609276 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/14d4c77a-4999-4d40-803a-80b4f77aad5d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-h2gwx\" (UID: \"14d4c77a-4999-4d40-803a-80b4f77aad5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:24.609450 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.609435 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/14d4c77a-4999-4d40-803a-80b4f77aad5d-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-h2gwx\" (UID: \"14d4c77a-4999-4d40-803a-80b4f77aad5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:24.609571 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.609558 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/14d4c77a-4999-4d40-803a-80b4f77aad5d-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-h2gwx\" (UID: \"14d4c77a-4999-4d40-803a-80b4f77aad5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:24.609686 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.609674 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d4c77a-4999-4d40-803a-80b4f77aad5d-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-h2gwx\" (UID: \"14d4c77a-4999-4d40-803a-80b4f77aad5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:24.609820 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.609808 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d4c77a-4999-4d40-803a-80b4f77aad5d-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-h2gwx\" (UID: \"14d4c77a-4999-4d40-803a-80b4f77aad5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:24.609953 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.609940 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/14d4c77a-4999-4d40-803a-80b4f77aad5d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-h2gwx\" (UID: \"14d4c77a-4999-4d40-803a-80b4f77aad5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:24.711104 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.711067 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d4c77a-4999-4d40-803a-80b4f77aad5d-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-h2gwx\" (UID: \"14d4c77a-4999-4d40-803a-80b4f77aad5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:24.711275 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.711122 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d4c77a-4999-4d40-803a-80b4f77aad5d-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-h2gwx\" (UID: \"14d4c77a-4999-4d40-803a-80b4f77aad5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:24.711275 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.711166 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/14d4c77a-4999-4d40-803a-80b4f77aad5d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-h2gwx\" (UID: \"14d4c77a-4999-4d40-803a-80b4f77aad5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:24.711275 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.711208 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7zzz\" (UniqueName: \"kubernetes.io/projected/14d4c77a-4999-4d40-803a-80b4f77aad5d-kube-api-access-m7zzz\") pod \"istiod-openshift-gateway-5677ccb49b-h2gwx\" (UID: \"14d4c77a-4999-4d40-803a-80b4f77aad5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:24.711275 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.711257 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/14d4c77a-4999-4d40-803a-80b4f77aad5d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-h2gwx\" (UID: \"14d4c77a-4999-4d40-803a-80b4f77aad5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:24.711533 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.711290 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/14d4c77a-4999-4d40-803a-80b4f77aad5d-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-h2gwx\" (UID: \"14d4c77a-4999-4d40-803a-80b4f77aad5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:24.711533 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.711337 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/14d4c77a-4999-4d40-803a-80b4f77aad5d-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-h2gwx\" (UID: \"14d4c77a-4999-4d40-803a-80b4f77aad5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:24.712165 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.712134 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/14d4c77a-4999-4d40-803a-80b4f77aad5d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-h2gwx\" (UID: \"14d4c77a-4999-4d40-803a-80b4f77aad5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:24.713684 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.713655 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/14d4c77a-4999-4d40-803a-80b4f77aad5d-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-h2gwx\" (UID: \"14d4c77a-4999-4d40-803a-80b4f77aad5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:24.713777 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.713684 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d4c77a-4999-4d40-803a-80b4f77aad5d-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-h2gwx\" (UID: \"14d4c77a-4999-4d40-803a-80b4f77aad5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:24.713777 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.713697 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/14d4c77a-4999-4d40-803a-80b4f77aad5d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-h2gwx\" (UID: \"14d4c77a-4999-4d40-803a-80b4f77aad5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:24.713849 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.713782 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/14d4c77a-4999-4d40-803a-80b4f77aad5d-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-h2gwx\" (UID: \"14d4c77a-4999-4d40-803a-80b4f77aad5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:24.722067 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.722049 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d4c77a-4999-4d40-803a-80b4f77aad5d-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-h2gwx\" (UID: \"14d4c77a-4999-4d40-803a-80b4f77aad5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:24.722153 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.722082 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7zzz\" (UniqueName: \"kubernetes.io/projected/14d4c77a-4999-4d40-803a-80b4f77aad5d-kube-api-access-m7zzz\") pod \"istiod-openshift-gateway-5677ccb49b-h2gwx\" (UID: \"14d4c77a-4999-4d40-803a-80b4f77aad5d\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:24.858367 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:24.858279 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:25.139177 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:25.139101 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx"] Sep 08 00:39:25.142111 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:39:25.142088 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d4c77a_4999_4d40_803a_80b4f77aad5d.slice/crio-26d3b0adf6b911673a53b7b54ce8a565e4705abe53e3acee4edb4aeb5c8a6339 WatchSource:0}: Error finding container 26d3b0adf6b911673a53b7b54ce8a565e4705abe53e3acee4edb4aeb5c8a6339: Status 404 returned error can't find the container with id 26d3b0adf6b911673a53b7b54ce8a565e4705abe53e3acee4edb4aeb5c8a6339 Sep 08 00:39:25.144394 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:25.144358 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:39:25.144483 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:25.144439 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:39:25.399168 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:25.399078 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" event={"ID":"14d4c77a-4999-4d40-803a-80b4f77aad5d","Type":"ContainerStarted","Data":"c85007924fb9fb5ddf188ec21d0c8ab55bef8e551438e3cb89fc66be656aadfa"} Sep 08 00:39:25.399168 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:25.399159 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" event={"ID":"14d4c77a-4999-4d40-803a-80b4f77aad5d","Type":"ContainerStarted","Data":"26d3b0adf6b911673a53b7b54ce8a565e4705abe53e3acee4edb4aeb5c8a6339"} Sep 08 00:39:25.399346 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:25.399218 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:25.424205 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:25.424155 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" podStartSLOduration=1.424137159 podStartE2EDuration="1.424137159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:39:25.422966909 +0000 UTC m=+379.118208072" watchObservedRunningTime="2026-09-08 00:39:25.424137159 +0000 UTC m=+379.119378318" Sep 08 00:39:26.406829 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.406796 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-h2gwx" Sep 08 00:39:26.454850 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.454810 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl"] Sep 08 00:39:26.455048 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.455022 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" podUID="f7d2f5f4-5e97-4a59-8834-e3b2798759bb" containerName="discovery" containerID="cri-o://f46f4f86445a0f55fbb07d8eb3cd53354779a38867ca74e28a33c60b4deae1cf" gracePeriod=30 Sep 08 00:39:26.696283 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.696259 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:39:26.830963 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.830934 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-cacerts\") pod \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " Sep 08 00:39:26.831158 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.830978 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-token\") pod \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " Sep 08 00:39:26.831158 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.831035 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-local-certs\") pod \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " Sep 08 00:39:26.831158 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.831065 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-kubeconfig\") pod \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " Sep 08 00:39:26.831158 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.831092 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-csr-dns-cert\") pod \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " Sep 08 00:39:26.831158 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.831139 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-csr-ca-configmap\") pod \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " Sep 08 00:39:26.831413 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.831164 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlxbk\" (UniqueName: \"kubernetes.io/projected/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-kube-api-access-zlxbk\") pod \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\" (UID: \"f7d2f5f4-5e97-4a59-8834-e3b2798759bb\") " Sep 08 00:39:26.831613 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.831568 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "f7d2f5f4-5e97-4a59-8834-e3b2798759bb" (UID: "f7d2f5f4-5e97-4a59-8834-e3b2798759bb"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:39:26.833399 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.833369 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-cacerts" (OuterVolumeSpecName: "cacerts") pod "f7d2f5f4-5e97-4a59-8834-e3b2798759bb" (UID: "f7d2f5f4-5e97-4a59-8834-e3b2798759bb"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 00:39:26.833497 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.833445 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "f7d2f5f4-5e97-4a59-8834-e3b2798759bb" (UID: "f7d2f5f4-5e97-4a59-8834-e3b2798759bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:39:26.833696 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.833665 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "f7d2f5f4-5e97-4a59-8834-e3b2798759bb" (UID: "f7d2f5f4-5e97-4a59-8834-e3b2798759bb"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 00:39:26.833696 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.833684 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "f7d2f5f4-5e97-4a59-8834-e3b2798759bb" (UID: "f7d2f5f4-5e97-4a59-8834-e3b2798759bb"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 00:39:26.833828 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.833741 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-local-certs" (OuterVolumeSpecName: "local-certs") pod "f7d2f5f4-5e97-4a59-8834-e3b2798759bb" (UID: "f7d2f5f4-5e97-4a59-8834-e3b2798759bb"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:39:26.833828 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.833774 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-kube-api-access-zlxbk" (OuterVolumeSpecName: "kube-api-access-zlxbk") pod "f7d2f5f4-5e97-4a59-8834-e3b2798759bb" (UID: "f7d2f5f4-5e97-4a59-8834-e3b2798759bb"). InnerVolumeSpecName "kube-api-access-zlxbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:39:26.932063 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.931977 2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-local-certs\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:39:26.932063 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.932024 2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-kubeconfig\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:39:26.932063 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.932036 2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-csr-dns-cert\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:39:26.932063 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.932045 2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-csr-ca-configmap\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:39:26.932063 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.932057 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlxbk\" (UniqueName: \"kubernetes.io/projected/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-kube-api-access-zlxbk\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:39:26.932363 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.932070 2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-cacerts\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:39:26.932363 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:26.932082 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7d2f5f4-5e97-4a59-8834-e3b2798759bb-istio-token\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:39:27.410408 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:27.410371 2577 generic.go:356] "Generic (PLEG): container finished" podID="f7d2f5f4-5e97-4a59-8834-e3b2798759bb" containerID="f46f4f86445a0f55fbb07d8eb3cd53354779a38867ca74e28a33c60b4deae1cf" exitCode=0 Sep 08 00:39:27.410796 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:27.410420 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" Sep 08 00:39:27.410796 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:27.410453 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" event={"ID":"f7d2f5f4-5e97-4a59-8834-e3b2798759bb","Type":"ContainerDied","Data":"f46f4f86445a0f55fbb07d8eb3cd53354779a38867ca74e28a33c60b4deae1cf"} Sep 08 00:39:27.410796 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:27.410503 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl" event={"ID":"f7d2f5f4-5e97-4a59-8834-e3b2798759bb","Type":"ContainerDied","Data":"45129468dfe2a7b543c9f82c78c2152c3c462699358ae537722c0a38e1348a56"} Sep 08 00:39:27.410796 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:27.410525 2577 scope.go:117] "RemoveContainer" containerID="f46f4f86445a0f55fbb07d8eb3cd53354779a38867ca74e28a33c60b4deae1cf" Sep 08 00:39:27.419447 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:27.419430 2577 scope.go:117] "RemoveContainer" containerID="f46f4f86445a0f55fbb07d8eb3cd53354779a38867ca74e28a33c60b4deae1cf" Sep 08 00:39:27.419726 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:39:27.419707 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f46f4f86445a0f55fbb07d8eb3cd53354779a38867ca74e28a33c60b4deae1cf\": container with ID starting with f46f4f86445a0f55fbb07d8eb3cd53354779a38867ca74e28a33c60b4deae1cf not found: ID does not exist" containerID="f46f4f86445a0f55fbb07d8eb3cd53354779a38867ca74e28a33c60b4deae1cf" Sep 08 00:39:27.419786 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:27.419738 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f46f4f86445a0f55fbb07d8eb3cd53354779a38867ca74e28a33c60b4deae1cf"} err="failed to get container status \"f46f4f86445a0f55fbb07d8eb3cd53354779a38867ca74e28a33c60b4deae1cf\": rpc error: code = NotFound desc = could not find container \"f46f4f86445a0f55fbb07d8eb3cd53354779a38867ca74e28a33c60b4deae1cf\": container with ID starting with f46f4f86445a0f55fbb07d8eb3cd53354779a38867ca74e28a33c60b4deae1cf not found: ID does not exist" Sep 08 00:39:27.431457 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:27.431420 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl"] Sep 08 00:39:27.436304 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:27.436285 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d4qcl"] Sep 08 00:39:28.790072 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:28.790037 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7d2f5f4-5e97-4a59-8834-e3b2798759bb" path="/var/lib/kubelet/pods/f7d2f5f4-5e97-4a59-8834-e3b2798759bb/volumes" Sep 08 00:39:34.396977 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:39:34.396947 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-55885fcb99-2fcml" Sep 08 00:41:33.319422 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.319377 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg"] Sep 08 00:41:33.320099 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.319781 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7d2f5f4-5e97-4a59-8834-e3b2798759bb" containerName="discovery" Sep 08 00:41:33.320099 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.319800 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7d2f5f4-5e97-4a59-8834-e3b2798759bb" containerName="discovery" Sep 08 00:41:33.320099 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.319872 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7d2f5f4-5e97-4a59-8834-e3b2798759bb" containerName="discovery" Sep 08 00:41:33.323022 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.322988 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:33.325821 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.325800 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 08 00:41:33.325945 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.325800 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 08 00:41:33.326942 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.326920 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cnbvb\"" Sep 08 00:41:33.327088 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.326990 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 08 00:41:33.330587 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.330567 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg"] Sep 08 00:41:33.427853 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.427821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdl2r\" (UniqueName: \"kubernetes.io/projected/e946943c-5a95-4267-b0bb-7b73f74f1ef9-kube-api-access-cdl2r\") pod \"facebook-opt-125m-simulated-kserve-89945f574-tbcrg\" (UID: \"e946943c-5a95-4267-b0bb-7b73f74f1ef9\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:33.428047 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.427859 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e946943c-5a95-4267-b0bb-7b73f74f1ef9-dshm\") pod \"facebook-opt-125m-simulated-kserve-89945f574-tbcrg\" (UID: \"e946943c-5a95-4267-b0bb-7b73f74f1ef9\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:33.428047 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.427879 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e946943c-5a95-4267-b0bb-7b73f74f1ef9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-89945f574-tbcrg\" (UID: \"e946943c-5a95-4267-b0bb-7b73f74f1ef9\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:33.428047 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.427898 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e946943c-5a95-4267-b0bb-7b73f74f1ef9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-89945f574-tbcrg\" (UID: \"e946943c-5a95-4267-b0bb-7b73f74f1ef9\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:33.428047 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.427988 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e946943c-5a95-4267-b0bb-7b73f74f1ef9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-89945f574-tbcrg\" (UID: \"e946943c-5a95-4267-b0bb-7b73f74f1ef9\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:33.428218 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.428061 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e946943c-5a95-4267-b0bb-7b73f74f1ef9-home\") pod \"facebook-opt-125m-simulated-kserve-89945f574-tbcrg\" (UID: \"e946943c-5a95-4267-b0bb-7b73f74f1ef9\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:33.528883 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.528848 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e946943c-5a95-4267-b0bb-7b73f74f1ef9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-89945f574-tbcrg\" (UID: \"e946943c-5a95-4267-b0bb-7b73f74f1ef9\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:33.529096 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.528896 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e946943c-5a95-4267-b0bb-7b73f74f1ef9-home\") pod \"facebook-opt-125m-simulated-kserve-89945f574-tbcrg\" (UID: \"e946943c-5a95-4267-b0bb-7b73f74f1ef9\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:33.529096 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.529029 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdl2r\" (UniqueName: \"kubernetes.io/projected/e946943c-5a95-4267-b0bb-7b73f74f1ef9-kube-api-access-cdl2r\") pod \"facebook-opt-125m-simulated-kserve-89945f574-tbcrg\" (UID: \"e946943c-5a95-4267-b0bb-7b73f74f1ef9\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:33.529096 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.529064 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e946943c-5a95-4267-b0bb-7b73f74f1ef9-dshm\") pod \"facebook-opt-125m-simulated-kserve-89945f574-tbcrg\" (UID: \"e946943c-5a95-4267-b0bb-7b73f74f1ef9\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:33.529096 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.529086 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e946943c-5a95-4267-b0bb-7b73f74f1ef9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-89945f574-tbcrg\" (UID: \"e946943c-5a95-4267-b0bb-7b73f74f1ef9\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:33.529324 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.529115 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e946943c-5a95-4267-b0bb-7b73f74f1ef9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-89945f574-tbcrg\" (UID: \"e946943c-5a95-4267-b0bb-7b73f74f1ef9\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:33.529379 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.529334 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e946943c-5a95-4267-b0bb-7b73f74f1ef9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-89945f574-tbcrg\" (UID: \"e946943c-5a95-4267-b0bb-7b73f74f1ef9\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:33.529418 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.529374 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e946943c-5a95-4267-b0bb-7b73f74f1ef9-home\") pod \"facebook-opt-125m-simulated-kserve-89945f574-tbcrg\" (UID: \"e946943c-5a95-4267-b0bb-7b73f74f1ef9\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:33.529418 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.529389 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e946943c-5a95-4267-b0bb-7b73f74f1ef9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-89945f574-tbcrg\" (UID: \"e946943c-5a95-4267-b0bb-7b73f74f1ef9\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:33.531396 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.531376 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e946943c-5a95-4267-b0bb-7b73f74f1ef9-dshm\") pod \"facebook-opt-125m-simulated-kserve-89945f574-tbcrg\" (UID: \"e946943c-5a95-4267-b0bb-7b73f74f1ef9\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:33.531574 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.531558 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e946943c-5a95-4267-b0bb-7b73f74f1ef9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-89945f574-tbcrg\" (UID: \"e946943c-5a95-4267-b0bb-7b73f74f1ef9\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:33.537282 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.537262 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdl2r\" (UniqueName: \"kubernetes.io/projected/e946943c-5a95-4267-b0bb-7b73f74f1ef9-kube-api-access-cdl2r\") pod \"facebook-opt-125m-simulated-kserve-89945f574-tbcrg\" (UID: \"e946943c-5a95-4267-b0bb-7b73f74f1ef9\") " pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:33.633706 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.633619 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:33.902177 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.901947 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg"] Sep 08 00:41:33.904658 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:41:33.904630 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode946943c_5a95_4267_b0bb_7b73f74f1ef9.slice/crio-35e1c1fb353fe7554e8810e39bbe62a5820d1ddfdc5011593c61fbc2df8adb64 WatchSource:0}: Error finding container 35e1c1fb353fe7554e8810e39bbe62a5820d1ddfdc5011593c61fbc2df8adb64: Status 404 returned error can't find the container with id 35e1c1fb353fe7554e8810e39bbe62a5820d1ddfdc5011593c61fbc2df8adb64 Sep 08 00:41:33.925263 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:33.925231 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" event={"ID":"e946943c-5a95-4267-b0bb-7b73f74f1ef9","Type":"ContainerStarted","Data":"35e1c1fb353fe7554e8810e39bbe62a5820d1ddfdc5011593c61fbc2df8adb64"} Sep 08 00:41:34.931686 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:34.931655 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" event={"ID":"e946943c-5a95-4267-b0bb-7b73f74f1ef9","Type":"ContainerStarted","Data":"78469eaed3f89529f15bd63e5666ed873e5c5ed1659f1073b5d763ea5858a80a"} Sep 08 00:41:34.949828 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:34.949780 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" podStartSLOduration=1.000448117 podStartE2EDuration="1.949764415s" totalImagesPullingTime="949.316298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:41:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:41:34.949140479 +0000 UTC m=+508.644381674" watchObservedRunningTime="2026-09-08 00:41:34.949764415 +0000 UTC m=+508.645005573" Sep 08 00:41:35.634093 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:35.634050 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:43.634167 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:43.634128 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:43.646558 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:43.646531 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:43.979177 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:43.979092 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-89945f574-tbcrg" Sep 08 00:41:45.836079 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:45.836044 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8"] Sep 08 00:41:45.839221 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:45.839200 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:45.841699 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:45.841677 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 08 00:41:45.848511 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:45.848490 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8"] Sep 08 00:41:45.925475 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:45.925436 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f89ca12-4299-43a7-950d-0d9ef5a0227b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8\" (UID: \"9f89ca12-4299-43a7-950d-0d9ef5a0227b\") " pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:45.925475 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:45.925476 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f89ca12-4299-43a7-950d-0d9ef5a0227b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8\" (UID: \"9f89ca12-4299-43a7-950d-0d9ef5a0227b\") " pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:45.925696 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:45.925534 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7bnn\" (UniqueName: \"kubernetes.io/projected/9f89ca12-4299-43a7-950d-0d9ef5a0227b-kube-api-access-l7bnn\") pod \"premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8\" (UID: \"9f89ca12-4299-43a7-950d-0d9ef5a0227b\") " pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:45.925696 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:45.925610 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f89ca12-4299-43a7-950d-0d9ef5a0227b-home\") pod \"premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8\" (UID: \"9f89ca12-4299-43a7-950d-0d9ef5a0227b\") " pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:45.925696 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:45.925658 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f89ca12-4299-43a7-950d-0d9ef5a0227b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8\" (UID: \"9f89ca12-4299-43a7-950d-0d9ef5a0227b\") " pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:45.925696 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:45.925688 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f89ca12-4299-43a7-950d-0d9ef5a0227b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8\" (UID: \"9f89ca12-4299-43a7-950d-0d9ef5a0227b\") " pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:46.027056 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:46.027020 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f89ca12-4299-43a7-950d-0d9ef5a0227b-home\") pod \"premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8\" (UID: \"9f89ca12-4299-43a7-950d-0d9ef5a0227b\") " pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:46.027056 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:46.027059 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f89ca12-4299-43a7-950d-0d9ef5a0227b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8\" (UID: \"9f89ca12-4299-43a7-950d-0d9ef5a0227b\") " pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:46.027312 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:46.027085 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f89ca12-4299-43a7-950d-0d9ef5a0227b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8\" (UID: \"9f89ca12-4299-43a7-950d-0d9ef5a0227b\") " pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:46.027312 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:46.027117 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f89ca12-4299-43a7-950d-0d9ef5a0227b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8\" (UID: \"9f89ca12-4299-43a7-950d-0d9ef5a0227b\") " pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:46.027312 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:46.027135 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f89ca12-4299-43a7-950d-0d9ef5a0227b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8\" (UID: \"9f89ca12-4299-43a7-950d-0d9ef5a0227b\") " pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:46.027312 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:46.027163 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7bnn\" (UniqueName: \"kubernetes.io/projected/9f89ca12-4299-43a7-950d-0d9ef5a0227b-kube-api-access-l7bnn\") pod \"premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8\" (UID: \"9f89ca12-4299-43a7-950d-0d9ef5a0227b\") " pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:46.027519 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:46.027475 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f89ca12-4299-43a7-950d-0d9ef5a0227b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8\" (UID: \"9f89ca12-4299-43a7-950d-0d9ef5a0227b\") " pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:46.027519 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:46.027473 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f89ca12-4299-43a7-950d-0d9ef5a0227b-home\") pod \"premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8\" (UID: \"9f89ca12-4299-43a7-950d-0d9ef5a0227b\") " pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:46.027600 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:46.027531 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f89ca12-4299-43a7-950d-0d9ef5a0227b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8\" (UID: \"9f89ca12-4299-43a7-950d-0d9ef5a0227b\") " pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:46.029418 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:46.029394 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f89ca12-4299-43a7-950d-0d9ef5a0227b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8\" (UID: \"9f89ca12-4299-43a7-950d-0d9ef5a0227b\") " pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:46.029671 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:46.029654 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f89ca12-4299-43a7-950d-0d9ef5a0227b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8\" (UID: \"9f89ca12-4299-43a7-950d-0d9ef5a0227b\") " pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:46.036572 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:46.036546 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7bnn\" (UniqueName: \"kubernetes.io/projected/9f89ca12-4299-43a7-950d-0d9ef5a0227b-kube-api-access-l7bnn\") pod \"premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8\" (UID: \"9f89ca12-4299-43a7-950d-0d9ef5a0227b\") " pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:46.148535 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:46.148503 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:46.416599 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:46.416573 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8"] Sep 08 00:41:46.418386 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:41:46.418362 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f89ca12_4299_43a7_950d_0d9ef5a0227b.slice/crio-cfe1e73219b2cb029ff3e15708d69bce446561e4f347539004c0ac4e363bfdf8 WatchSource:0}: Error finding container cfe1e73219b2cb029ff3e15708d69bce446561e4f347539004c0ac4e363bfdf8: Status 404 returned error can't find the container with id cfe1e73219b2cb029ff3e15708d69bce446561e4f347539004c0ac4e363bfdf8 Sep 08 00:41:46.983652 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:46.983615 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" event={"ID":"9f89ca12-4299-43a7-950d-0d9ef5a0227b","Type":"ContainerStarted","Data":"420085a9e24713239487ca8de3d55d3fb55d185170d151b00ecbd558e1ec50c0"} Sep 08 00:41:46.984045 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:46.983665 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" event={"ID":"9f89ca12-4299-43a7-950d-0d9ef5a0227b","Type":"ContainerStarted","Data":"cfe1e73219b2cb029ff3e15708d69bce446561e4f347539004c0ac4e363bfdf8"} Sep 08 00:41:47.001946 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:47.001898 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" podStartSLOduration=1.831707084 podStartE2EDuration="2.00188234s" totalImagesPullingTime="170.175256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:41:47.000583644 +0000 UTC m=+520.695824840" watchObservedRunningTime="2026-09-08 00:41:47.00188234 +0000 UTC m=+520.697123498" Sep 08 00:41:47.149252 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:47.149212 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:56.148906 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:56.148861 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:56.161420 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:56.161391 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:41:57.038473 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:41:57.038442 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8" Sep 08 00:43:06.693293 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:06.693263 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9nb9x_a2e160ae-d75d-4959-a5ab-d238b4113736/console-operator/1.log" Sep 08 00:43:06.693768 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:06.693266 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9nb9x_a2e160ae-d75d-4959-a5ab-d238b4113736/console-operator/1.log" Sep 08 00:43:33.990761 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:33.990683 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl"] Sep 08 00:43:33.993959 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:33.993944 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl" Sep 08 00:43:33.996455 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:33.996433 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-31f424\"" Sep 08 00:43:33.996554 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:33.996515 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-31f424-dockercfg-47c4l\"" Sep 08 00:43:34.003018 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:34.002972 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl"] Sep 08 00:43:34.027911 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:34.027878 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z"] Sep 08 00:43:34.031128 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:34.031102 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z" Sep 08 00:43:34.042302 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:34.042275 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z"] Sep 08 00:43:34.097241 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:34.097203 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f84a3d-f6fb-4067-bf08-629980db649b-plugins-config-volume\") pod \"payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z\" (UID: \"80f84a3d-f6fb-4067-bf08-629980db649b\") " pod="openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z" Sep 08 00:43:34.097401 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:34.097250 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0654cf5-8dd6-4180-b7f1-5b83cfc598b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl\" (UID: \"f0654cf5-8dd6-4180-b7f1-5b83cfc598b2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl" Sep 08 00:43:34.097401 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:34.097367 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7ckl\" (UniqueName: \"kubernetes.io/projected/f0654cf5-8dd6-4180-b7f1-5b83cfc598b2-kube-api-access-b7ckl\") pod \"payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl\" (UID: \"f0654cf5-8dd6-4180-b7f1-5b83cfc598b2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl" Sep 08 00:43:34.097505 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:34.097400 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl4cl\" (UniqueName: \"kubernetes.io/projected/80f84a3d-f6fb-4067-bf08-629980db649b-kube-api-access-cl4cl\") pod \"payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z\" (UID: \"80f84a3d-f6fb-4067-bf08-629980db649b\") " pod="openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z" Sep 08 00:43:34.198083 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:34.198047 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f84a3d-f6fb-4067-bf08-629980db649b-plugins-config-volume\") pod \"payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z\" (UID: \"80f84a3d-f6fb-4067-bf08-629980db649b\") " pod="openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z" Sep 08 00:43:34.198263 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:34.198102 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0654cf5-8dd6-4180-b7f1-5b83cfc598b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl\" (UID: \"f0654cf5-8dd6-4180-b7f1-5b83cfc598b2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl" Sep 08 00:43:34.198263 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:34.198172 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7ckl\" (UniqueName: \"kubernetes.io/projected/f0654cf5-8dd6-4180-b7f1-5b83cfc598b2-kube-api-access-b7ckl\") pod \"payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl\" (UID: \"f0654cf5-8dd6-4180-b7f1-5b83cfc598b2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl" Sep 08 00:43:34.198263 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:34.198199 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl4cl\" (UniqueName: \"kubernetes.io/projected/80f84a3d-f6fb-4067-bf08-629980db649b-kube-api-access-cl4cl\") pod \"payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z\" (UID: \"80f84a3d-f6fb-4067-bf08-629980db649b\") " pod="openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z" Sep 08 00:43:34.198840 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:34.198814 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f84a3d-f6fb-4067-bf08-629980db649b-plugins-config-volume\") pod \"payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z\" (UID: \"80f84a3d-f6fb-4067-bf08-629980db649b\") " pod="openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z" Sep 08 00:43:34.198934 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:34.198819 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0654cf5-8dd6-4180-b7f1-5b83cfc598b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl\" (UID: \"f0654cf5-8dd6-4180-b7f1-5b83cfc598b2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl" Sep 08 00:43:34.206601 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:34.206581 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7ckl\" (UniqueName: \"kubernetes.io/projected/f0654cf5-8dd6-4180-b7f1-5b83cfc598b2-kube-api-access-b7ckl\") pod \"payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl\" (UID: \"f0654cf5-8dd6-4180-b7f1-5b83cfc598b2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl" Sep 08 00:43:34.206696 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:34.206626 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl4cl\" (UniqueName: \"kubernetes.io/projected/80f84a3d-f6fb-4067-bf08-629980db649b-kube-api-access-cl4cl\") pod \"payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z\" (UID: \"80f84a3d-f6fb-4067-bf08-629980db649b\") " pod="openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z" Sep 08 00:43:34.304339 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:34.304309 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl" Sep 08 00:43:34.341579 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:34.341550 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z" Sep 08 00:43:34.579470 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:34.579445 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl"] Sep 08 00:43:34.581632 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:43:34.581607 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0654cf5_8dd6_4180_b7f1_5b83cfc598b2.slice/crio-78e9d14d35dcb37504731a5856ae9917b13727a6f291c37f723f83b2e8e4fd07 WatchSource:0}: Error finding container 78e9d14d35dcb37504731a5856ae9917b13727a6f291c37f723f83b2e8e4fd07: Status 404 returned error can't find the container with id 78e9d14d35dcb37504731a5856ae9917b13727a6f291c37f723f83b2e8e4fd07 Sep 08 00:43:34.583435 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:34.583417 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 00:43:34.617892 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:34.617863 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z"] Sep 08 00:43:34.621549 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:43:34.621525 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f84a3d_f6fb_4067_bf08_629980db649b.slice/crio-d63a92c3ba724a85b60819cec7dc1e0d8fb09dcfdfa77c1da58e6fc4861fc087 WatchSource:0}: Error finding container d63a92c3ba724a85b60819cec7dc1e0d8fb09dcfdfa77c1da58e6fc4861fc087: Status 404 returned error can't find the container with id d63a92c3ba724a85b60819cec7dc1e0d8fb09dcfdfa77c1da58e6fc4861fc087 Sep 08 00:43:35.443720 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:35.443684 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl" event={"ID":"f0654cf5-8dd6-4180-b7f1-5b83cfc598b2","Type":"ContainerStarted","Data":"78e9d14d35dcb37504731a5856ae9917b13727a6f291c37f723f83b2e8e4fd07"} Sep 08 00:43:35.445812 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:35.445783 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z" event={"ID":"80f84a3d-f6fb-4067-bf08-629980db649b","Type":"ContainerStarted","Data":"d63a92c3ba724a85b60819cec7dc1e0d8fb09dcfdfa77c1da58e6fc4861fc087"} Sep 08 00:43:36.453272 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:36.453072 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl" event={"ID":"f0654cf5-8dd6-4180-b7f1-5b83cfc598b2","Type":"ContainerStarted","Data":"b1a9004f2e51277206d946aaa1a07a74ba0da2026edeb34aab742040e55e9d4f"} Sep 08 00:43:36.453272 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:36.453138 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl" Sep 08 00:43:36.454815 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:36.454788 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z" event={"ID":"80f84a3d-f6fb-4067-bf08-629980db649b","Type":"ContainerStarted","Data":"6d7f1c4dfc50a873b46d7a0d81ef718580fe45c7032578c9b1dd1322dbd52bfe"} Sep 08 00:43:36.454941 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:36.454914 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z" Sep 08 00:43:36.473637 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:36.473594 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl" podStartSLOduration=1.6786724259999999 podStartE2EDuration="3.473577227s" totalImagesPullingTime="1.794904801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:43:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:43:36.472651318 +0000 UTC m=+630.167892477" watchObservedRunningTime="2026-09-08 00:43:36.473577227 +0000 UTC m=+630.168818388" Sep 08 00:43:36.492032 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:36.491971 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z" podStartSLOduration=0.743007191 podStartE2EDuration="2.491958124s" totalImagesPullingTime="1.748950933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:43:36.491433191 +0000 UTC m=+630.186674351" watchObservedRunningTime="2026-09-08 00:43:36.491958124 +0000 UTC m=+630.187199633" Sep 08 00:43:47.461745 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:47.461713 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl" Sep 08 00:43:47.462190 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:47.461781 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z" Sep 08 00:43:49.781480 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.781447 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw"] Sep 08 00:43:49.784644 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.784619 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.787773 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.787748 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e3896b-openshift-default-dockercfg-fg964\"" Sep 08 00:43:49.796219 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.796196 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw"] Sep 08 00:43:49.837460 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.837430 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-workload-certs\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.837624 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.837495 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w6nh\" (UniqueName: \"kubernetes.io/projected/009b85b2-8630-42b6-875a-ed084b0593b6-kube-api-access-4w6nh\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.837624 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.837553 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-istio-data\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.837624 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.837605 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-workload-socket\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.837624 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.837621 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-credential-socket\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.837775 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.837638 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009b85b2-8630-42b6-875a-ed084b0593b6-istio-token\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.837775 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.837750 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-istio-envoy\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.837848 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.837780 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009b85b2-8630-42b6-875a-ed084b0593b6-istiod-ca-cert\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.837848 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.837804 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009b85b2-8630-42b6-875a-ed084b0593b6-istio-podinfo\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.938337 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.938306 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-istio-envoy\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.938521 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.938345 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009b85b2-8630-42b6-875a-ed084b0593b6-istiod-ca-cert\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.938521 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.938371 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009b85b2-8630-42b6-875a-ed084b0593b6-istio-podinfo\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.938521 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.938401 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-workload-certs\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.938521 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.938454 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w6nh\" (UniqueName: \"kubernetes.io/projected/009b85b2-8630-42b6-875a-ed084b0593b6-kube-api-access-4w6nh\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.938521 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.938481 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-istio-data\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.938521 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.938517 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-workload-socket\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.938915 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.938858 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-workload-certs\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.938915 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.938890 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-workload-socket\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.939118 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.938859 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-credential-socket\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.939118 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.938952 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009b85b2-8630-42b6-875a-ed084b0593b6-istio-token\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.939118 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.939022 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-istio-data\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.939269 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.939144 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009b85b2-8630-42b6-875a-ed084b0593b6-istiod-ca-cert\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.939269 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.939195 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-credential-socket\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.940685 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.940665 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-istio-envoy\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.940811 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.940793 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009b85b2-8630-42b6-875a-ed084b0593b6-istio-podinfo\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.947042 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.947021 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009b85b2-8630-42b6-875a-ed084b0593b6-istio-token\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:49.947231 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:49.947210 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w6nh\" (UniqueName: \"kubernetes.io/projected/009b85b2-8630-42b6-875a-ed084b0593b6-kube-api-access-4w6nh\") pod \"e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:50.097379 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:50.097281 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:50.369703 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:50.369676 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw"] Sep 08 00:43:50.371566 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:43:50.371525 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009b85b2_8630_42b6_875a_ed084b0593b6.slice/crio-29f0831c28aa32b2f7be9796a3206cb38194de0575894855e4ab660816cef295 WatchSource:0}: Error finding container 29f0831c28aa32b2f7be9796a3206cb38194de0575894855e4ab660816cef295: Status 404 returned error can't find the container with id 29f0831c28aa32b2f7be9796a3206cb38194de0575894855e4ab660816cef295 Sep 08 00:43:50.519930 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:50.519889 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" event={"ID":"009b85b2-8630-42b6-875a-ed084b0593b6","Type":"ContainerStarted","Data":"29f0831c28aa32b2f7be9796a3206cb38194de0575894855e4ab660816cef295"} Sep 08 00:43:52.893729 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:52.893687 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:43:52.894037 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:52.893760 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:43:52.894037 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:52.893793 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:43:53.537323 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:53.537284 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" event={"ID":"009b85b2-8630-42b6-875a-ed084b0593b6","Type":"ContainerStarted","Data":"9420baf779f185608d5fd580a71cff614abaa18104011e43385b7a6024514b12"} Sep 08 00:43:53.560436 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:53.560388 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" podStartSLOduration=2.040756576 podStartE2EDuration="4.560372326s" totalImagesPullingTime="2.51961575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:43:53.558147962 +0000 UTC m=+647.253389117" watchObservedRunningTime="2026-09-08 00:43:53.560372326 +0000 UTC m=+647.255613486" Sep 08 00:43:54.097588 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:54.097556 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:54.102567 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:54.102544 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:54.541725 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:54.541683 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:54.542631 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:54.542610 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:43:58.272038 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.271990 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h"] Sep 08 00:43:58.275342 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.275324 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h" Sep 08 00:43:58.278191 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.278171 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e3896b-dockercfg-2jsrk\"" Sep 08 00:43:58.278303 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.278202 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e3896b\"" Sep 08 00:43:58.287913 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.287891 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h"] Sep 08 00:43:58.360657 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.360624 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b"] Sep 08 00:43:58.363883 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.363863 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b" Sep 08 00:43:58.374531 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.374507 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b"] Sep 08 00:43:58.416602 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.416566 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q2sq\" (UniqueName: \"kubernetes.io/projected/6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb-kube-api-access-6q2sq\") pod \"payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h\" (UID: \"6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h" Sep 08 00:43:58.416741 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.416617 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h\" (UID: \"6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h" Sep 08 00:43:58.518009 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.517960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q2sq\" (UniqueName: \"kubernetes.io/projected/6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb-kube-api-access-6q2sq\") pod \"payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h\" (UID: \"6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h" Sep 08 00:43:58.518009 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.518013 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c065d6b7-06a5-4f9a-ba41-9cfb027bfa40-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b\" (UID: \"c065d6b7-06a5-4f9a-ba41-9cfb027bfa40\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b" Sep 08 00:43:58.518201 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.518047 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h\" (UID: \"6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h" Sep 08 00:43:58.518201 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.518064 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf55k\" (UniqueName: \"kubernetes.io/projected/c065d6b7-06a5-4f9a-ba41-9cfb027bfa40-kube-api-access-mf55k\") pod \"payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b\" (UID: \"c065d6b7-06a5-4f9a-ba41-9cfb027bfa40\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b" Sep 08 00:43:58.518715 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.518695 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h\" (UID: \"6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h" Sep 08 00:43:58.530962 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.530935 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q2sq\" (UniqueName: \"kubernetes.io/projected/6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb-kube-api-access-6q2sq\") pod \"payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h\" (UID: \"6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h" Sep 08 00:43:58.585454 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.585429 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h" Sep 08 00:43:58.618596 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.618565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf55k\" (UniqueName: \"kubernetes.io/projected/c065d6b7-06a5-4f9a-ba41-9cfb027bfa40-kube-api-access-mf55k\") pod \"payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b\" (UID: \"c065d6b7-06a5-4f9a-ba41-9cfb027bfa40\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b" Sep 08 00:43:58.618717 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.618650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c065d6b7-06a5-4f9a-ba41-9cfb027bfa40-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b\" (UID: \"c065d6b7-06a5-4f9a-ba41-9cfb027bfa40\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b" Sep 08 00:43:58.619273 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.619254 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c065d6b7-06a5-4f9a-ba41-9cfb027bfa40-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b\" (UID: \"c065d6b7-06a5-4f9a-ba41-9cfb027bfa40\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b" Sep 08 00:43:58.626535 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.626512 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf55k\" (UniqueName: \"kubernetes.io/projected/c065d6b7-06a5-4f9a-ba41-9cfb027bfa40-kube-api-access-mf55k\") pod \"payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b\" (UID: \"c065d6b7-06a5-4f9a-ba41-9cfb027bfa40\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b" Sep 08 00:43:58.676878 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.676844 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b" Sep 08 00:43:58.856872 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.856847 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h"] Sep 08 00:43:58.858521 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:43:58.858496 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a8aa5a5_8d5e_4f7f_b731_9827e7798bbb.slice/crio-56066bb9112f175284e17f1c90d46aaed30e188c3f1570083fd3bd213f4d0220 WatchSource:0}: Error finding container 56066bb9112f175284e17f1c90d46aaed30e188c3f1570083fd3bd213f4d0220: Status 404 returned error can't find the container with id 56066bb9112f175284e17f1c90d46aaed30e188c3f1570083fd3bd213f4d0220 Sep 08 00:43:58.953083 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:58.953050 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b"] Sep 08 00:43:58.956097 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:43:58.956068 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc065d6b7_06a5_4f9a_ba41_9cfb027bfa40.slice/crio-0c33f50da2fe9106b58b489ef03b5f6d9d9eb0e8ce442982dc38486a8b24dd73 WatchSource:0}: Error finding container 0c33f50da2fe9106b58b489ef03b5f6d9d9eb0e8ce442982dc38486a8b24dd73: Status 404 returned error can't find the container with id 0c33f50da2fe9106b58b489ef03b5f6d9d9eb0e8ce442982dc38486a8b24dd73 Sep 08 00:43:59.567565 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:59.567524 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b" event={"ID":"c065d6b7-06a5-4f9a-ba41-9cfb027bfa40","Type":"ContainerStarted","Data":"b8cf3267fdd22bbcab221364aaa47fa4a95384aabc34cd15e5f5376bfe4911a5"} Sep 08 00:43:59.568022 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:59.567686 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b" Sep 08 00:43:59.568022 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:59.567725 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b" event={"ID":"c065d6b7-06a5-4f9a-ba41-9cfb027bfa40","Type":"ContainerStarted","Data":"0c33f50da2fe9106b58b489ef03b5f6d9d9eb0e8ce442982dc38486a8b24dd73"} Sep 08 00:43:59.569198 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:59.569174 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h" event={"ID":"6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb","Type":"ContainerStarted","Data":"2a458f7324694c1a108155aaf54db279124327c78a77d3729ec7e22a0e705b29"} Sep 08 00:43:59.569299 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:59.569212 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h" event={"ID":"6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb","Type":"ContainerStarted","Data":"56066bb9112f175284e17f1c90d46aaed30e188c3f1570083fd3bd213f4d0220"} Sep 08 00:43:59.569345 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:59.569307 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h" Sep 08 00:43:59.586586 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:59.586538 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b" podStartSLOduration=1.586520759 podStartE2EDuration="1.586520759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:43:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:43:59.585135297 +0000 UTC m=+653.280376456" watchObservedRunningTime="2026-09-08 00:43:59.586520759 +0000 UTC m=+653.281761918" Sep 08 00:43:59.599604 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:43:59.599555 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h" podStartSLOduration=1.599538941 podStartE2EDuration="1.599538941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:43:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:43:59.598490301 +0000 UTC m=+653.293731462" watchObservedRunningTime="2026-09-08 00:43:59.599538941 +0000 UTC m=+653.294780101" Sep 08 00:44:01.617206 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.617173 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j"] Sep 08 00:44:01.620660 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.620639 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:01.623164 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.623145 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-4a8d93c0-kserve-self-signed-certs\"" Sep 08 00:44:01.631830 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.631807 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j"] Sep 08 00:44:01.748952 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.748911 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-home\") pod \"e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:01.748952 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.748952 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-model-cache\") pod \"e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:01.749227 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.748983 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtpcz\" (UniqueName: \"kubernetes.io/projected/6c3c0f13-0cc9-4ba7-9988-6422214e154f-kube-api-access-dtpcz\") pod \"e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:01.749227 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.749137 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-dshm\") pod \"e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:01.749227 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.749170 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-tmp-dir\") pod \"e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:01.749336 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.749261 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c3c0f13-0cc9-4ba7-9988-6422214e154f-tls-certs\") pod \"e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:01.849861 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.849819 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-dshm\") pod \"e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:01.849861 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.849860 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-tmp-dir\") pod \"e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:01.850123 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.849889 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c3c0f13-0cc9-4ba7-9988-6422214e154f-tls-certs\") pod \"e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:01.850123 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.850042 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-home\") pod \"e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:01.850123 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.850083 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-model-cache\") pod \"e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:01.850278 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.850203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtpcz\" (UniqueName: \"kubernetes.io/projected/6c3c0f13-0cc9-4ba7-9988-6422214e154f-kube-api-access-dtpcz\") pod \"e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:01.850384 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.850359 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-tmp-dir\") pod \"e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:01.850455 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.850384 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-home\") pod \"e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:01.850455 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.850427 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-model-cache\") pod \"e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:01.852043 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.852021 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-dshm\") pod \"e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:01.852289 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.852272 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c3c0f13-0cc9-4ba7-9988-6422214e154f-tls-certs\") pod \"e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:01.857281 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.857259 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtpcz\" (UniqueName: \"kubernetes.io/projected/6c3c0f13-0cc9-4ba7-9988-6422214e154f-kube-api-access-dtpcz\") pod \"e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:01.929497 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:01.929419 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:02.197845 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:02.197822 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j"] Sep 08 00:44:02.199550 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:44:02.199526 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c3c0f13_0cc9_4ba7_9988_6422214e154f.slice/crio-04570d728867fb955804adfaf1cbc2deb7524d4ed31211475a4becf3e7abfafa WatchSource:0}: Error finding container 04570d728867fb955804adfaf1cbc2deb7524d4ed31211475a4becf3e7abfafa: Status 404 returned error can't find the container with id 04570d728867fb955804adfaf1cbc2deb7524d4ed31211475a4becf3e7abfafa Sep 08 00:44:02.582526 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:02.582492 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" event={"ID":"6c3c0f13-0cc9-4ba7-9988-6422214e154f","Type":"ContainerStarted","Data":"7a285185c7015ab59e9238758bbc0996541a195cb6fabfbd24a54a6ea7adbe47"} Sep 08 00:44:02.582696 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:02.582536 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" event={"ID":"6c3c0f13-0cc9-4ba7-9988-6422214e154f","Type":"ContainerStarted","Data":"04570d728867fb955804adfaf1cbc2deb7524d4ed31211475a4becf3e7abfafa"} Sep 08 00:44:02.602259 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:02.602203 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" podStartSLOduration=1.6021864890000002 podStartE2EDuration="1.602186489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:44:02.599609819 +0000 UTC m=+656.294850978" watchObservedRunningTime="2026-09-08 00:44:02.602186489 +0000 UTC m=+656.297427649" Sep 08 00:44:02.930080 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:02.929982 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:02.961765 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:02.961733 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b0b8262-96cw2"] Sep 08 00:44:02.965247 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:02.965227 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b0b8262-96cw2" Sep 08 00:44:02.967475 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:02.967457 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 00:44:02.967569 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:02.967462 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dqw2\"" Sep 08 00:44:02.971674 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:02.971651 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b0b8262-96cw2"] Sep 08 00:44:03.060725 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:03.060685 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f0562c-4bfe-46c9-8ac8-6f9743585b21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3b0b8262-96cw2\" (UID: \"78f0562c-4bfe-46c9-8ac8-6f9743585b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b0b8262-96cw2" Sep 08 00:44:03.162168 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:03.162134 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f0562c-4bfe-46c9-8ac8-6f9743585b21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3b0b8262-96cw2\" (UID: \"78f0562c-4bfe-46c9-8ac8-6f9743585b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b0b8262-96cw2" Sep 08 00:44:03.162751 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:03.162733 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f0562c-4bfe-46c9-8ac8-6f9743585b21-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3b0b8262-96cw2\" (UID: \"78f0562c-4bfe-46c9-8ac8-6f9743585b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b0b8262-96cw2" Sep 08 00:44:03.277799 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:03.277778 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b0b8262-96cw2" Sep 08 00:44:03.546107 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:03.546084 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b0b8262-96cw2"] Sep 08 00:44:03.548116 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:44:03.548089 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f0562c_4bfe_46c9_8ac8_6f9743585b21.slice/crio-fdb9b88f2f9accfb45c344821a074a589bb53e250b4ad8c6a33757d3f7115476 WatchSource:0}: Error finding container fdb9b88f2f9accfb45c344821a074a589bb53e250b4ad8c6a33757d3f7115476: Status 404 returned error can't find the container with id fdb9b88f2f9accfb45c344821a074a589bb53e250b4ad8c6a33757d3f7115476 Sep 08 00:44:03.587542 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:03.587517 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b0b8262-96cw2" event={"ID":"78f0562c-4bfe-46c9-8ac8-6f9743585b21","Type":"ContainerStarted","Data":"fdb9b88f2f9accfb45c344821a074a589bb53e250b4ad8c6a33757d3f7115476"} Sep 08 00:44:06.608122 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:06.608086 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b0b8262-96cw2" event={"ID":"78f0562c-4bfe-46c9-8ac8-6f9743585b21","Type":"ContainerStarted","Data":"4d3ccdf0ac7035fa6ce47cfd76095e37074d4c89f803c002595d8b3c81422b52"} Sep 08 00:44:06.625830 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:06.625782 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b0b8262-96cw2" podStartSLOduration=1.977902694 podStartE2EDuration="4.625767198s" totalImagesPullingTime="2.647864504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:44:06.624066197 +0000 UTC m=+660.319307361" watchObservedRunningTime="2026-09-08 00:44:06.625767198 +0000 UTC m=+660.321008357" Sep 08 00:44:07.614178 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:07.614137 2577 generic.go:356] "Generic (PLEG): container finished" podID="78f0562c-4bfe-46c9-8ac8-6f9743585b21" containerID="4d3ccdf0ac7035fa6ce47cfd76095e37074d4c89f803c002595d8b3c81422b52" exitCode=0 Sep 08 00:44:07.614587 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:07.614235 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b0b8262-96cw2" event={"ID":"78f0562c-4bfe-46c9-8ac8-6f9743585b21","Type":"ContainerDied","Data":"4d3ccdf0ac7035fa6ce47cfd76095e37074d4c89f803c002595d8b3c81422b52"} Sep 08 00:44:08.745548 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:08.745524 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b0b8262-96cw2" Sep 08 00:44:08.907970 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:08.907877 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f0562c-4bfe-46c9-8ac8-6f9743585b21-maas-api-service-ca\") pod \"78f0562c-4bfe-46c9-8ac8-6f9743585b21\" (UID: \"78f0562c-4bfe-46c9-8ac8-6f9743585b21\") " Sep 08 00:44:08.908313 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:08.908277 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78f0562c-4bfe-46c9-8ac8-6f9743585b21-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "78f0562c-4bfe-46c9-8ac8-6f9743585b21" (UID: "78f0562c-4bfe-46c9-8ac8-6f9743585b21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:44:09.008623 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:09.008593 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78f0562c-4bfe-46c9-8ac8-6f9743585b21-maas-api-service-ca\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:44:09.623663 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:09.623628 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b0b8262-96cw2" Sep 08 00:44:09.623663 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:09.623650 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b0b8262-96cw2" event={"ID":"78f0562c-4bfe-46c9-8ac8-6f9743585b21","Type":"ContainerDied","Data":"fdb9b88f2f9accfb45c344821a074a589bb53e250b4ad8c6a33757d3f7115476"} Sep 08 00:44:09.623867 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:09.623690 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdb9b88f2f9accfb45c344821a074a589bb53e250b4ad8c6a33757d3f7115476" Sep 08 00:44:10.576114 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:10.576078 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h" Sep 08 00:44:10.576520 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:10.576148 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b" Sep 08 00:44:11.930633 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:11.930591 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:11.943459 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:11.943431 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:12.651115 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:12.651080 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:44:18.791397 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:18.791363 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b0b8262-96cw2"] Sep 08 00:44:18.792182 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:18.792159 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3b0b8262-96cw2"] Sep 08 00:44:20.790342 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:20.790312 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78f0562c-4bfe-46c9-8ac8-6f9743585b21" path="/var/lib/kubelet/pods/78f0562c-4bfe-46c9-8ac8-6f9743585b21/volumes" Sep 08 00:44:21.683715 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:21.683681 2577 generic.go:356] "Generic (PLEG): container finished" podID="009b85b2-8630-42b6-875a-ed084b0593b6" containerID="9420baf779f185608d5fd580a71cff614abaa18104011e43385b7a6024514b12" exitCode=0 Sep 08 00:44:21.683884 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:21.683734 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" event={"ID":"009b85b2-8630-42b6-875a-ed084b0593b6","Type":"ContainerDied","Data":"9420baf779f185608d5fd580a71cff614abaa18104011e43385b7a6024514b12"} Sep 08 00:44:21.684122 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:21.684107 2577 scope.go:117] "RemoveContainer" containerID="9420baf779f185608d5fd580a71cff614abaa18104011e43385b7a6024514b12" Sep 08 00:44:21.685172 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:21.685143 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:44:21.685660 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:21.685618 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:44:21.685918 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:21.685893 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:44:22.690030 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:22.689982 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" event={"ID":"009b85b2-8630-42b6-875a-ed084b0593b6","Type":"ContainerStarted","Data":"87e3d4274cd843ccd69491bf24b7eb0e80cd2d500a227a6461d37795a13534cd"} Sep 08 00:44:22.906782 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:22.906750 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb"] Sep 08 00:44:22.907125 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:22.907112 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78f0562c-4bfe-46c9-8ac8-6f9743585b21" containerName="revoke-keys" Sep 08 00:44:22.907180 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:22.907127 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="78f0562c-4bfe-46c9-8ac8-6f9743585b21" containerName="revoke-keys" Sep 08 00:44:22.907215 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:22.907184 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="78f0562c-4bfe-46c9-8ac8-6f9743585b21" containerName="revoke-keys" Sep 08 00:44:22.908991 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:22.908970 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb" Sep 08 00:44:22.911497 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:22.911472 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-3790db3d-dockercfg-lc468\"" Sep 08 00:44:22.911740 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:22.911717 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-3790db3d\"" Sep 08 00:44:22.922090 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:22.922055 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb"] Sep 08 00:44:22.933539 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:22.933493 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26d95\" (UniqueName: \"kubernetes.io/projected/c1910f9c-a7ac-4cf8-87dc-4a56398e3a85-kube-api-access-26d95\") pod \"payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb\" (UID: \"c1910f9c-a7ac-4cf8-87dc-4a56398e3a85\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb" Sep 08 00:44:22.933844 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:22.933809 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1910f9c-a7ac-4cf8-87dc-4a56398e3a85-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb\" (UID: \"c1910f9c-a7ac-4cf8-87dc-4a56398e3a85\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb" Sep 08 00:44:23.034758 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:23.034722 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26d95\" (UniqueName: \"kubernetes.io/projected/c1910f9c-a7ac-4cf8-87dc-4a56398e3a85-kube-api-access-26d95\") pod \"payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb\" (UID: \"c1910f9c-a7ac-4cf8-87dc-4a56398e3a85\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb" Sep 08 00:44:23.034961 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:23.034775 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1910f9c-a7ac-4cf8-87dc-4a56398e3a85-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb\" (UID: \"c1910f9c-a7ac-4cf8-87dc-4a56398e3a85\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb" Sep 08 00:44:23.035556 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:23.035531 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1910f9c-a7ac-4cf8-87dc-4a56398e3a85-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb\" (UID: \"c1910f9c-a7ac-4cf8-87dc-4a56398e3a85\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb" Sep 08 00:44:23.043350 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:23.043322 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26d95\" (UniqueName: \"kubernetes.io/projected/c1910f9c-a7ac-4cf8-87dc-4a56398e3a85-kube-api-access-26d95\") pod \"payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb\" (UID: \"c1910f9c-a7ac-4cf8-87dc-4a56398e3a85\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb" Sep 08 00:44:23.098446 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:23.098411 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:44:23.099651 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:23.099613 2577 patch_prober.go:28] interesting pod/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Sep 08 00:44:23.099875 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:23.099841 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" podUID="009b85b2-8630-42b6-875a-ed084b0593b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Sep 08 00:44:23.227459 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:23.227426 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb" Sep 08 00:44:23.497294 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:23.497192 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb"] Sep 08 00:44:23.700816 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:23.700772 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb" event={"ID":"c1910f9c-a7ac-4cf8-87dc-4a56398e3a85","Type":"ContainerStarted","Data":"219d62da4e210f140ce3fe9ac2070eddba59f313252e99a122f229a931b71525"} Sep 08 00:44:23.701344 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:23.700837 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb" event={"ID":"c1910f9c-a7ac-4cf8-87dc-4a56398e3a85","Type":"ContainerStarted","Data":"e7067b7de24b8c6eac39d9f92701161800c62e369cac916c3d194f83e5b7454c"} Sep 08 00:44:23.701344 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:23.700884 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb" Sep 08 00:44:23.720364 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:23.720300 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb" podStartSLOduration=1.720277549 podStartE2EDuration="1.720277549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:44:23.717972729 +0000 UTC m=+677.413213891" watchObservedRunningTime="2026-09-08 00:44:23.720277549 +0000 UTC m=+677.415518712" Sep 08 00:44:24.097773 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:24.097733 2577 patch_prober.go:28] interesting pod/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Sep 08 00:44:24.097957 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:24.097807 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" podUID="009b85b2-8630-42b6-875a-ed084b0593b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Sep 08 00:44:24.541962 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:24.541929 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:44:25.098819 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:25.098775 2577 patch_prober.go:28] interesting pod/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Sep 08 00:44:25.099323 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:25.098851 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" podUID="009b85b2-8630-42b6-875a-ed084b0593b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Sep 08 00:44:26.098312 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:26.098274 2577 patch_prober.go:28] interesting pod/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Sep 08 00:44:26.098504 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:26.098343 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" podUID="009b85b2-8630-42b6-875a-ed084b0593b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Sep 08 00:44:27.098390 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:27.098347 2577 patch_prober.go:28] interesting pod/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Sep 08 00:44:27.098881 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:27.098418 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" podUID="009b85b2-8630-42b6-875a-ed084b0593b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Sep 08 00:44:28.098691 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:28.098649 2577 patch_prober.go:28] interesting pod/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Sep 08 00:44:28.099328 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:28.099295 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" podUID="009b85b2-8630-42b6-875a-ed084b0593b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Sep 08 00:44:29.101795 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:29.101765 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:44:29.102796 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:29.102775 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:44:35.717430 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:35.717397 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb" Sep 08 00:44:47.154279 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:47.154241 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3790db3d-frzh4"] Sep 08 00:44:47.161991 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:47.161968 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3790db3d-frzh4" Sep 08 00:44:47.164959 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:47.164936 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 00:44:47.165570 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:47.165547 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dqw2\"" Sep 08 00:44:47.166978 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:47.166958 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3790db3d-frzh4"] Sep 08 00:44:47.248210 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:47.248179 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6206da33-c6ff-483c-bc18-cef0573c6c4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3790db3d-frzh4\" (UID: \"6206da33-c6ff-483c-bc18-cef0573c6c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3790db3d-frzh4" Sep 08 00:44:47.349372 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:47.349335 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6206da33-c6ff-483c-bc18-cef0573c6c4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3790db3d-frzh4\" (UID: \"6206da33-c6ff-483c-bc18-cef0573c6c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3790db3d-frzh4" Sep 08 00:44:47.349959 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:47.349940 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6206da33-c6ff-483c-bc18-cef0573c6c4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3790db3d-frzh4\" (UID: \"6206da33-c6ff-483c-bc18-cef0573c6c4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3790db3d-frzh4" Sep 08 00:44:47.473309 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:47.473231 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3790db3d-frzh4" Sep 08 00:44:47.740499 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:47.740472 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3790db3d-frzh4"] Sep 08 00:44:47.745704 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:44:47.745670 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6206da33_c6ff_483c_bc18_cef0573c6c4f.slice/crio-69e8e59e2d8093e4410449ebbadee8ba72bbd4ca8a352402c469e2d9b2e03428 WatchSource:0}: Error finding container 69e8e59e2d8093e4410449ebbadee8ba72bbd4ca8a352402c469e2d9b2e03428: Status 404 returned error can't find the container with id 69e8e59e2d8093e4410449ebbadee8ba72bbd4ca8a352402c469e2d9b2e03428 Sep 08 00:44:47.821172 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:47.821135 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3790db3d-frzh4" event={"ID":"6206da33-c6ff-483c-bc18-cef0573c6c4f","Type":"ContainerStarted","Data":"69e8e59e2d8093e4410449ebbadee8ba72bbd4ca8a352402c469e2d9b2e03428"} Sep 08 00:44:48.392520 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:44:48.392494 2577 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6206da33_c6ff_483c_bc18_cef0573c6c4f.slice/crio-conmon-5fa8e448d96467042edb55156c527bab5614deaf0e765a3c1e1efb35e6318c12.scope\": RecentStats: unable to find data in memory cache]" Sep 08 00:44:48.826570 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:48.826536 2577 generic.go:356] "Generic (PLEG): container finished" podID="6206da33-c6ff-483c-bc18-cef0573c6c4f" containerID="5fa8e448d96467042edb55156c527bab5614deaf0e765a3c1e1efb35e6318c12" exitCode=0 Sep 08 00:44:48.826749 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:48.826600 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3790db3d-frzh4" event={"ID":"6206da33-c6ff-483c-bc18-cef0573c6c4f","Type":"ContainerDied","Data":"5fa8e448d96467042edb55156c527bab5614deaf0e765a3c1e1efb35e6318c12"} Sep 08 00:44:49.956666 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:49.956642 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3790db3d-frzh4" Sep 08 00:44:50.071682 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:50.071647 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6206da33-c6ff-483c-bc18-cef0573c6c4f-maas-api-service-ca\") pod \"6206da33-c6ff-483c-bc18-cef0573c6c4f\" (UID: \"6206da33-c6ff-483c-bc18-cef0573c6c4f\") " Sep 08 00:44:50.072035 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:50.071984 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6206da33-c6ff-483c-bc18-cef0573c6c4f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6206da33-c6ff-483c-bc18-cef0573c6c4f" (UID: "6206da33-c6ff-483c-bc18-cef0573c6c4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:44:50.172818 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:50.172731 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6206da33-c6ff-483c-bc18-cef0573c6c4f-maas-api-service-ca\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:44:50.836955 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:50.836927 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3790db3d-frzh4" Sep 08 00:44:50.836955 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:50.836946 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3790db3d-frzh4" event={"ID":"6206da33-c6ff-483c-bc18-cef0573c6c4f","Type":"ContainerDied","Data":"69e8e59e2d8093e4410449ebbadee8ba72bbd4ca8a352402c469e2d9b2e03428"} Sep 08 00:44:50.837170 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:50.837016 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69e8e59e2d8093e4410449ebbadee8ba72bbd4ca8a352402c469e2d9b2e03428" Sep 08 00:44:53.653706 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:53.653665 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb"] Sep 08 00:44:53.654155 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:53.653911 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb" podUID="c1910f9c-a7ac-4cf8-87dc-4a56398e3a85" containerName="payload-pre-processing" containerID="cri-o://219d62da4e210f140ce3fe9ac2070eddba59f313252e99a122f229a931b71525" gracePeriod=30 Sep 08 00:44:53.833268 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:53.833237 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3790db3d-frzh4"] Sep 08 00:44:53.838193 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:53.838168 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3790db3d-frzh4"] Sep 08 00:44:53.852470 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:53.852445 2577 generic.go:356] "Generic (PLEG): container finished" podID="c1910f9c-a7ac-4cf8-87dc-4a56398e3a85" containerID="219d62da4e210f140ce3fe9ac2070eddba59f313252e99a122f229a931b71525" exitCode=0 Sep 08 00:44:53.852624 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:53.852528 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb" event={"ID":"c1910f9c-a7ac-4cf8-87dc-4a56398e3a85","Type":"ContainerDied","Data":"219d62da4e210f140ce3fe9ac2070eddba59f313252e99a122f229a931b71525"} Sep 08 00:44:53.900766 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:53.900744 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb" Sep 08 00:44:54.007918 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:54.007833 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1910f9c-a7ac-4cf8-87dc-4a56398e3a85-plugins-config-volume\") pod \"c1910f9c-a7ac-4cf8-87dc-4a56398e3a85\" (UID: \"c1910f9c-a7ac-4cf8-87dc-4a56398e3a85\") " Sep 08 00:44:54.007918 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:54.007886 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26d95\" (UniqueName: \"kubernetes.io/projected/c1910f9c-a7ac-4cf8-87dc-4a56398e3a85-kube-api-access-26d95\") pod \"c1910f9c-a7ac-4cf8-87dc-4a56398e3a85\" (UID: \"c1910f9c-a7ac-4cf8-87dc-4a56398e3a85\") " Sep 08 00:44:54.008282 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:54.008258 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1910f9c-a7ac-4cf8-87dc-4a56398e3a85-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c1910f9c-a7ac-4cf8-87dc-4a56398e3a85" (UID: "c1910f9c-a7ac-4cf8-87dc-4a56398e3a85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:44:54.009938 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:54.009912 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1910f9c-a7ac-4cf8-87dc-4a56398e3a85-kube-api-access-26d95" (OuterVolumeSpecName: "kube-api-access-26d95") pod "c1910f9c-a7ac-4cf8-87dc-4a56398e3a85" (UID: "c1910f9c-a7ac-4cf8-87dc-4a56398e3a85"). InnerVolumeSpecName "kube-api-access-26d95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:44:54.108692 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:54.108659 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1910f9c-a7ac-4cf8-87dc-4a56398e3a85-plugins-config-volume\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:44:54.108692 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:54.108688 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26d95\" (UniqueName: \"kubernetes.io/projected/c1910f9c-a7ac-4cf8-87dc-4a56398e3a85-kube-api-access-26d95\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:44:54.792492 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:54.792453 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6206da33-c6ff-483c-bc18-cef0573c6c4f" path="/var/lib/kubelet/pods/6206da33-c6ff-483c-bc18-cef0573c6c4f/volumes" Sep 08 00:44:54.858616 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:54.858591 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb" Sep 08 00:44:54.858765 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:54.858586 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb" event={"ID":"c1910f9c-a7ac-4cf8-87dc-4a56398e3a85","Type":"ContainerDied","Data":"e7067b7de24b8c6eac39d9f92701161800c62e369cac916c3d194f83e5b7454c"} Sep 08 00:44:54.858765 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:54.858699 2577 scope.go:117] "RemoveContainer" containerID="219d62da4e210f140ce3fe9ac2070eddba59f313252e99a122f229a931b71525" Sep 08 00:44:54.876254 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:54.876227 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb"] Sep 08 00:44:54.879369 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:54.879349 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-3790db3d-65966cdb78-4qwxb"] Sep 08 00:44:56.790686 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:56.790651 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1910f9c-a7ac-4cf8-87dc-4a56398e3a85" path="/var/lib/kubelet/pods/c1910f9c-a7ac-4cf8-87dc-4a56398e3a85/volumes" Sep 08 00:44:59.300555 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.300481 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm"] Sep 08 00:44:59.300955 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.300798 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6206da33-c6ff-483c-bc18-cef0573c6c4f" containerName="revoke-keys" Sep 08 00:44:59.300955 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.300810 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6206da33-c6ff-483c-bc18-cef0573c6c4f" containerName="revoke-keys" Sep 08 00:44:59.300955 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.300834 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1910f9c-a7ac-4cf8-87dc-4a56398e3a85" containerName="payload-pre-processing" Sep 08 00:44:59.300955 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.300841 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1910f9c-a7ac-4cf8-87dc-4a56398e3a85" containerName="payload-pre-processing" Sep 08 00:44:59.300955 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.300911 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1910f9c-a7ac-4cf8-87dc-4a56398e3a85" containerName="payload-pre-processing" Sep 08 00:44:59.300955 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.300920 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6206da33-c6ff-483c-bc18-cef0573c6c4f" containerName="revoke-keys" Sep 08 00:44:59.306429 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.306395 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm" Sep 08 00:44:59.307627 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.307591 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz"] Sep 08 00:44:59.309709 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.309683 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f80520f6\"" Sep 08 00:44:59.309851 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.309836 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f80520f6-dockercfg-6w7qv\"" Sep 08 00:44:59.312354 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.312331 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz" Sep 08 00:44:59.316179 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.316131 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm"] Sep 08 00:44:59.319649 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.319627 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz"] Sep 08 00:44:59.352822 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.352796 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0c77dd-42bc-43e3-bf19-3d34f3910617-plugins-config-volume\") pod \"payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz\" (UID: \"9f0c77dd-42bc-43e3-bf19-3d34f3910617\") " pod="openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz" Sep 08 00:44:59.352949 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.352847 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bb5571-1ca6-46c2-9466-5caf9e15649f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm\" (UID: \"35bb5571-1ca6-46c2-9466-5caf9e15649f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm" Sep 08 00:44:59.352949 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.352895 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4s9n\" (UniqueName: \"kubernetes.io/projected/35bb5571-1ca6-46c2-9466-5caf9e15649f-kube-api-access-t4s9n\") pod \"payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm\" (UID: \"35bb5571-1ca6-46c2-9466-5caf9e15649f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm" Sep 08 00:44:59.352949 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.352916 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2chnj\" (UniqueName: \"kubernetes.io/projected/9f0c77dd-42bc-43e3-bf19-3d34f3910617-kube-api-access-2chnj\") pod \"payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz\" (UID: \"9f0c77dd-42bc-43e3-bf19-3d34f3910617\") " pod="openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz" Sep 08 00:44:59.453613 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.453578 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2chnj\" (UniqueName: \"kubernetes.io/projected/9f0c77dd-42bc-43e3-bf19-3d34f3910617-kube-api-access-2chnj\") pod \"payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz\" (UID: \"9f0c77dd-42bc-43e3-bf19-3d34f3910617\") " pod="openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz" Sep 08 00:44:59.453780 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.453629 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0c77dd-42bc-43e3-bf19-3d34f3910617-plugins-config-volume\") pod \"payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz\" (UID: \"9f0c77dd-42bc-43e3-bf19-3d34f3910617\") " pod="openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz" Sep 08 00:44:59.453780 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.453681 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bb5571-1ca6-46c2-9466-5caf9e15649f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm\" (UID: \"35bb5571-1ca6-46c2-9466-5caf9e15649f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm" Sep 08 00:44:59.453780 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.453724 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4s9n\" (UniqueName: \"kubernetes.io/projected/35bb5571-1ca6-46c2-9466-5caf9e15649f-kube-api-access-t4s9n\") pod \"payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm\" (UID: \"35bb5571-1ca6-46c2-9466-5caf9e15649f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm" Sep 08 00:44:59.454476 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.454455 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bb5571-1ca6-46c2-9466-5caf9e15649f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm\" (UID: \"35bb5571-1ca6-46c2-9466-5caf9e15649f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm" Sep 08 00:44:59.454527 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.454457 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0c77dd-42bc-43e3-bf19-3d34f3910617-plugins-config-volume\") pod \"payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz\" (UID: \"9f0c77dd-42bc-43e3-bf19-3d34f3910617\") " pod="openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz" Sep 08 00:44:59.461193 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.461161 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2chnj\" (UniqueName: \"kubernetes.io/projected/9f0c77dd-42bc-43e3-bf19-3d34f3910617-kube-api-access-2chnj\") pod \"payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz\" (UID: \"9f0c77dd-42bc-43e3-bf19-3d34f3910617\") " pod="openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz" Sep 08 00:44:59.461296 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.461208 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4s9n\" (UniqueName: \"kubernetes.io/projected/35bb5571-1ca6-46c2-9466-5caf9e15649f-kube-api-access-t4s9n\") pod \"payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm\" (UID: \"35bb5571-1ca6-46c2-9466-5caf9e15649f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm" Sep 08 00:44:59.620715 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.620631 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm" Sep 08 00:44:59.627692 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.627672 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz" Sep 08 00:44:59.896039 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.896016 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm"] Sep 08 00:44:59.898195 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:44:59.898172 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35bb5571_1ca6_46c2_9466_5caf9e15649f.slice/crio-dfdd1948452bfb95c3737127d6ac0edc2fe4ba6685428ac2899c56e713d5866a WatchSource:0}: Error finding container dfdd1948452bfb95c3737127d6ac0edc2fe4ba6685428ac2899c56e713d5866a: Status 404 returned error can't find the container with id dfdd1948452bfb95c3737127d6ac0edc2fe4ba6685428ac2899c56e713d5866a Sep 08 00:44:59.904024 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:44:59.903984 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz"] Sep 08 00:44:59.906614 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:44:59.906593 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0c77dd_42bc_43e3_bf19_3d34f3910617.slice/crio-35fbb3a46f2b7ae3d2146391e6e3baedee84e86b59a6eef45af4de1d965f5d57 WatchSource:0}: Error finding container 35fbb3a46f2b7ae3d2146391e6e3baedee84e86b59a6eef45af4de1d965f5d57: Status 404 returned error can't find the container with id 35fbb3a46f2b7ae3d2146391e6e3baedee84e86b59a6eef45af4de1d965f5d57 Sep 08 00:45:00.137609 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.137566 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813805-zpx59"] Sep 08 00:45:00.141235 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.141176 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813805-zpx59" Sep 08 00:45:00.141503 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.141477 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-31f424-29813805-vm89z"] Sep 08 00:45:00.143615 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.143591 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dqw2\"" Sep 08 00:45:00.144908 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.144889 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1cf84f-29813805-dcmdg"] Sep 08 00:45:00.145070 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.145054 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-31f424-29813805-vm89z" Sep 08 00:45:00.148140 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.148083 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1cf84f-29813805-dcmdg" Sep 08 00:45:00.151212 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.150986 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813805-zpx59"] Sep 08 00:45:00.153867 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.153843 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-31f424-29813805-vm89z"] Sep 08 00:45:00.157337 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.157213 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813805-zpx59" Sep 08 00:45:00.167759 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.167734 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-31f424-29813805-vm89z" Sep 08 00:45:00.170364 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.170328 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1cf84f-29813805-dcmdg"] Sep 08 00:45:00.187019 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.186975 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1cf84f-29813805-dcmdg" Sep 08 00:45:00.457924 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.457899 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813805-zpx59"] Sep 08 00:45:00.459371 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:45:00.459341 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b3a14e_862b_408e_aa2d_e176e0bb7147.slice/crio-0065b8e9609b8113415d2b739b54740361b2092879ade57721d343de22f61f55 WatchSource:0}: Error finding container 0065b8e9609b8113415d2b739b54740361b2092879ade57721d343de22f61f55: Status 404 returned error can't find the container with id 0065b8e9609b8113415d2b739b54740361b2092879ade57721d343de22f61f55 Sep 08 00:45:00.489507 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.489481 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-31f424-29813805-vm89z"] Sep 08 00:45:00.491965 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.491943 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1cf84f-29813805-dcmdg"] Sep 08 00:45:00.498341 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:45:00.498320 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d6e639_0bb4_4032_94f4_6c039b71a3d1.slice/crio-2b98e602a4bbb5f5dac05613b716f25d8372553daf20888e9f3e37b6fa33724c WatchSource:0}: Error finding container 2b98e602a4bbb5f5dac05613b716f25d8372553daf20888e9f3e37b6fa33724c: Status 404 returned error can't find the container with id 2b98e602a4bbb5f5dac05613b716f25d8372553daf20888e9f3e37b6fa33724c Sep 08 00:45:00.498524 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:45:00.498508 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d439106_d1eb_43a8_8da0_ac9e7c47c086.slice/crio-c152c45d6b7f574eafe281b221d0ac91ab7355e5e53e1134f9f36096cfddd078 WatchSource:0}: Error finding container c152c45d6b7f574eafe281b221d0ac91ab7355e5e53e1134f9f36096cfddd078: Status 404 returned error can't find the container with id c152c45d6b7f574eafe281b221d0ac91ab7355e5e53e1134f9f36096cfddd078 Sep 08 00:45:00.889134 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.888988 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1cf84f-29813805-dcmdg" event={"ID":"01d6e639-0bb4-4032-94f4-6c039b71a3d1","Type":"ContainerStarted","Data":"1db335abf7eeae7936921517bf1d7aba724b9f5036e8bccffbdf81927d2e3af5"} Sep 08 00:45:00.889134 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.889085 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1cf84f-29813805-dcmdg" event={"ID":"01d6e639-0bb4-4032-94f4-6c039b71a3d1","Type":"ContainerStarted","Data":"2b98e602a4bbb5f5dac05613b716f25d8372553daf20888e9f3e37b6fa33724c"} Sep 08 00:45:00.890917 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.890831 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813805-zpx59" event={"ID":"c2b3a14e-862b-408e-aa2d-e176e0bb7147","Type":"ContainerStarted","Data":"01e69dfdc4ae15b6cca6861fc7e7348b5d6b8cb7122e13c28df52bdbd1fda92c"} Sep 08 00:45:00.890917 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.890881 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813805-zpx59" event={"ID":"c2b3a14e-862b-408e-aa2d-e176e0bb7147","Type":"ContainerStarted","Data":"0065b8e9609b8113415d2b739b54740361b2092879ade57721d343de22f61f55"} Sep 08 00:45:00.892295 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.892266 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-31f424-29813805-vm89z" event={"ID":"3d439106-d1eb-43a8-8da0-ac9e7c47c086","Type":"ContainerStarted","Data":"c152c45d6b7f574eafe281b221d0ac91ab7355e5e53e1134f9f36096cfddd078"} Sep 08 00:45:00.894047 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.894026 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz" event={"ID":"9f0c77dd-42bc-43e3-bf19-3d34f3910617","Type":"ContainerStarted","Data":"5d79732a70873ecff63028738d9bd8018d50c5f497f3ee5540bdb450228a3a24"} Sep 08 00:45:00.894146 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.894069 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz" event={"ID":"9f0c77dd-42bc-43e3-bf19-3d34f3910617","Type":"ContainerStarted","Data":"35fbb3a46f2b7ae3d2146391e6e3baedee84e86b59a6eef45af4de1d965f5d57"} Sep 08 00:45:00.894146 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.894102 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz" Sep 08 00:45:00.895608 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.895589 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm" event={"ID":"35bb5571-1ca6-46c2-9466-5caf9e15649f","Type":"ContainerStarted","Data":"876b54f8f6ebec1e959862666506e497bf787fea7c826edc1084870730213a91"} Sep 08 00:45:00.895704 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.895620 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm" event={"ID":"35bb5571-1ca6-46c2-9466-5caf9e15649f","Type":"ContainerStarted","Data":"dfdd1948452bfb95c3737127d6ac0edc2fe4ba6685428ac2899c56e713d5866a"} Sep 08 00:45:00.895704 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.895651 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm" Sep 08 00:45:00.909374 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.909332 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1cf84f-29813805-dcmdg" podStartSLOduration=0.909318546 podStartE2EDuration="909.318546ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:45:00.907146898 +0000 UTC m=+714.602388051" watchObservedRunningTime="2026-09-08 00:45:00.909318546 +0000 UTC m=+714.604559755" Sep 08 00:45:00.926894 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.926834 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm" podStartSLOduration=1.9268161 podStartE2EDuration="1.9268161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:45:00.925325633 +0000 UTC m=+714.620566816" watchObservedRunningTime="2026-09-08 00:45:00.9268161 +0000 UTC m=+714.622057263" Sep 08 00:45:00.943025 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.942971 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz" podStartSLOduration=1.942957184 podStartE2EDuration="1.942957184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:45:00.941140714 +0000 UTC m=+714.636381886" watchObservedRunningTime="2026-09-08 00:45:00.942957184 +0000 UTC m=+714.638198385" Sep 08 00:45:00.956119 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:00.956072 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813805-zpx59" podStartSLOduration=0.956058163 podStartE2EDuration="956.058163ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:45:00.954624823 +0000 UTC m=+714.649865983" watchObservedRunningTime="2026-09-08 00:45:00.956058163 +0000 UTC m=+714.651299608" Sep 08 00:45:01.902181 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:01.902150 2577 generic.go:356] "Generic (PLEG): container finished" podID="01d6e639-0bb4-4032-94f4-6c039b71a3d1" containerID="1db335abf7eeae7936921517bf1d7aba724b9f5036e8bccffbdf81927d2e3af5" exitCode=0 Sep 08 00:45:01.902593 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:01.902237 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1cf84f-29813805-dcmdg" event={"ID":"01d6e639-0bb4-4032-94f4-6c039b71a3d1","Type":"ContainerDied","Data":"1db335abf7eeae7936921517bf1d7aba724b9f5036e8bccffbdf81927d2e3af5"} Sep 08 00:45:01.903696 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:01.903668 2577 generic.go:356] "Generic (PLEG): container finished" podID="c2b3a14e-862b-408e-aa2d-e176e0bb7147" containerID="01e69dfdc4ae15b6cca6861fc7e7348b5d6b8cb7122e13c28df52bdbd1fda92c" exitCode=0 Sep 08 00:45:01.903806 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:01.903748 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813805-zpx59" event={"ID":"c2b3a14e-862b-408e-aa2d-e176e0bb7147","Type":"ContainerDied","Data":"01e69dfdc4ae15b6cca6861fc7e7348b5d6b8cb7122e13c28df52bdbd1fda92c"} Sep 08 00:45:01.905267 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:01.905244 2577 generic.go:356] "Generic (PLEG): container finished" podID="3d439106-d1eb-43a8-8da0-ac9e7c47c086" containerID="8f5ba08711b6ebb476b9ef34873c5925f37584d94dfc1ce1bd0fef6053bb057a" exitCode=0 Sep 08 00:45:01.905401 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:01.905297 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-31f424-29813805-vm89z" event={"ID":"3d439106-d1eb-43a8-8da0-ac9e7c47c086","Type":"ContainerDied","Data":"8f5ba08711b6ebb476b9ef34873c5925f37584d94dfc1ce1bd0fef6053bb057a"} Sep 08 00:45:03.054816 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:03.054793 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1cf84f-29813805-dcmdg" Sep 08 00:45:03.089827 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:03.089780 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-31f424-29813805-vm89z" Sep 08 00:45:03.093070 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:03.093041 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813805-zpx59" Sep 08 00:45:03.917004 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:03.916976 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-31f424-29813805-vm89z" Sep 08 00:45:03.917186 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:03.916974 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-31f424-29813805-vm89z" event={"ID":"3d439106-d1eb-43a8-8da0-ac9e7c47c086","Type":"ContainerDied","Data":"c152c45d6b7f574eafe281b221d0ac91ab7355e5e53e1134f9f36096cfddd078"} Sep 08 00:45:03.917186 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:03.917135 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c152c45d6b7f574eafe281b221d0ac91ab7355e5e53e1134f9f36096cfddd078" Sep 08 00:45:03.918428 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:03.918412 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1cf84f-29813805-dcmdg" Sep 08 00:45:03.918535 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:03.918440 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1cf84f-29813805-dcmdg" event={"ID":"01d6e639-0bb4-4032-94f4-6c039b71a3d1","Type":"ContainerDied","Data":"2b98e602a4bbb5f5dac05613b716f25d8372553daf20888e9f3e37b6fa33724c"} Sep 08 00:45:03.918535 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:03.918486 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b98e602a4bbb5f5dac05613b716f25d8372553daf20888e9f3e37b6fa33724c" Sep 08 00:45:03.919850 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:03.919829 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813805-zpx59" event={"ID":"c2b3a14e-862b-408e-aa2d-e176e0bb7147","Type":"ContainerDied","Data":"0065b8e9609b8113415d2b739b54740361b2092879ade57721d343de22f61f55"} Sep 08 00:45:03.919960 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:03.919866 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813805-zpx59" Sep 08 00:45:03.919960 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:03.919868 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0065b8e9609b8113415d2b739b54740361b2092879ade57721d343de22f61f55" Sep 08 00:45:11.907162 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:11.907130 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm" Sep 08 00:45:11.907752 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:11.907733 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz" Sep 08 00:45:22.984036 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:22.983982 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f80520f6-hn5gj"] Sep 08 00:45:22.984453 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:22.984338 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01d6e639-0bb4-4032-94f4-6c039b71a3d1" containerName="cleanup" Sep 08 00:45:22.984453 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:22.984349 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="01d6e639-0bb4-4032-94f4-6c039b71a3d1" containerName="cleanup" Sep 08 00:45:22.984453 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:22.984361 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d439106-d1eb-43a8-8da0-ac9e7c47c086" containerName="cleanup" Sep 08 00:45:22.984453 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:22.984367 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d439106-d1eb-43a8-8da0-ac9e7c47c086" containerName="cleanup" Sep 08 00:45:22.984453 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:22.984384 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2b3a14e-862b-408e-aa2d-e176e0bb7147" containerName="cleanup" Sep 08 00:45:22.984453 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:22.984390 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2b3a14e-862b-408e-aa2d-e176e0bb7147" containerName="cleanup" Sep 08 00:45:22.984453 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:22.984445 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2b3a14e-862b-408e-aa2d-e176e0bb7147" containerName="cleanup" Sep 08 00:45:22.984453 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:22.984455 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="01d6e639-0bb4-4032-94f4-6c039b71a3d1" containerName="cleanup" Sep 08 00:45:22.984696 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:22.984460 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d439106-d1eb-43a8-8da0-ac9e7c47c086" containerName="cleanup" Sep 08 00:45:22.988745 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:22.988727 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f80520f6-hn5gj" Sep 08 00:45:22.992100 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:22.992076 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 00:45:22.992326 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:22.992113 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dqw2\"" Sep 08 00:45:22.993678 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:22.993649 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f80520f6-hn5gj"] Sep 08 00:45:23.067328 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:23.067287 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb84fcc-a190-4210-9407-cb481344f93c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f80520f6-hn5gj\" (UID: \"4eb84fcc-a190-4210-9407-cb481344f93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f80520f6-hn5gj" Sep 08 00:45:23.168510 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:23.168467 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb84fcc-a190-4210-9407-cb481344f93c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f80520f6-hn5gj\" (UID: \"4eb84fcc-a190-4210-9407-cb481344f93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f80520f6-hn5gj" Sep 08 00:45:23.169092 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:23.169070 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb84fcc-a190-4210-9407-cb481344f93c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f80520f6-hn5gj\" (UID: \"4eb84fcc-a190-4210-9407-cb481344f93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f80520f6-hn5gj" Sep 08 00:45:23.300718 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:23.300687 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f80520f6-hn5gj" Sep 08 00:45:23.568084 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:23.568057 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f80520f6-hn5gj"] Sep 08 00:45:23.570146 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:45:23.570121 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb84fcc_a190_4210_9407_cb481344f93c.slice/crio-48702b3d0d53adbd034a479c7fa3ee2993dc24a2730a64002a0ada3e06162027 WatchSource:0}: Error finding container 48702b3d0d53adbd034a479c7fa3ee2993dc24a2730a64002a0ada3e06162027: Status 404 returned error can't find the container with id 48702b3d0d53adbd034a479c7fa3ee2993dc24a2730a64002a0ada3e06162027 Sep 08 00:45:24.029100 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:24.029066 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f80520f6-hn5gj" event={"ID":"4eb84fcc-a190-4210-9407-cb481344f93c","Type":"ContainerStarted","Data":"057975b39394563756a1f7943e493bc2b860f7da0c0c5369763d0640d391977e"} Sep 08 00:45:24.029474 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:24.029128 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f80520f6-hn5gj" event={"ID":"4eb84fcc-a190-4210-9407-cb481344f93c","Type":"ContainerStarted","Data":"48702b3d0d53adbd034a479c7fa3ee2993dc24a2730a64002a0ada3e06162027"} Sep 08 00:45:24.047972 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:24.047921 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f80520f6-hn5gj" podStartSLOduration=2.04790473 podStartE2EDuration="2.04790473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:45:24.0462459 +0000 UTC m=+737.741487059" watchObservedRunningTime="2026-09-08 00:45:24.04790473 +0000 UTC m=+737.743145889" Sep 08 00:45:25.035243 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:25.035205 2577 generic.go:356] "Generic (PLEG): container finished" podID="4eb84fcc-a190-4210-9407-cb481344f93c" containerID="057975b39394563756a1f7943e493bc2b860f7da0c0c5369763d0640d391977e" exitCode=0 Sep 08 00:45:25.035698 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:25.035262 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f80520f6-hn5gj" event={"ID":"4eb84fcc-a190-4210-9407-cb481344f93c","Type":"ContainerDied","Data":"057975b39394563756a1f7943e493bc2b860f7da0c0c5369763d0640d391977e"} Sep 08 00:45:26.173611 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:26.173590 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f80520f6-hn5gj" Sep 08 00:45:26.294955 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:26.294876 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb84fcc-a190-4210-9407-cb481344f93c-maas-api-service-ca\") pod \"4eb84fcc-a190-4210-9407-cb481344f93c\" (UID: \"4eb84fcc-a190-4210-9407-cb481344f93c\") " Sep 08 00:45:26.295232 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:26.295207 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb84fcc-a190-4210-9407-cb481344f93c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4eb84fcc-a190-4210-9407-cb481344f93c" (UID: "4eb84fcc-a190-4210-9407-cb481344f93c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:45:26.396429 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:26.396397 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eb84fcc-a190-4210-9407-cb481344f93c-maas-api-service-ca\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:45:27.047241 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:27.047205 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f80520f6-hn5gj" Sep 08 00:45:27.047432 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:27.047240 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f80520f6-hn5gj" event={"ID":"4eb84fcc-a190-4210-9407-cb481344f93c","Type":"ContainerDied","Data":"48702b3d0d53adbd034a479c7fa3ee2993dc24a2730a64002a0ada3e06162027"} Sep 08 00:45:27.047432 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:27.047285 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48702b3d0d53adbd034a479c7fa3ee2993dc24a2730a64002a0ada3e06162027" Sep 08 00:45:29.053893 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:29.053863 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz"] Sep 08 00:45:29.054375 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:29.054089 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz" podUID="9f0c77dd-42bc-43e3-bf19-3d34f3910617" containerName="payload-processing" containerID="cri-o://5d79732a70873ecff63028738d9bd8018d50c5f497f3ee5540bdb450228a3a24" gracePeriod=30 Sep 08 00:45:29.060353 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:29.060325 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm"] Sep 08 00:45:29.060567 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:29.060543 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm" podUID="35bb5571-1ca6-46c2-9466-5caf9e15649f" containerName="payload-pre-processing" containerID="cri-o://876b54f8f6ebec1e959862666506e497bf787fea7c826edc1084870730213a91" gracePeriod=30 Sep 08 00:45:29.234311 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:29.234279 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f80520f6-hn5gj"] Sep 08 00:45:29.239596 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:29.239558 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f80520f6-hn5gj"] Sep 08 00:45:29.335024 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:29.334987 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz" Sep 08 00:45:29.337813 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:29.337787 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm" Sep 08 00:45:29.425928 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:29.425898 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2chnj\" (UniqueName: \"kubernetes.io/projected/9f0c77dd-42bc-43e3-bf19-3d34f3910617-kube-api-access-2chnj\") pod \"9f0c77dd-42bc-43e3-bf19-3d34f3910617\" (UID: \"9f0c77dd-42bc-43e3-bf19-3d34f3910617\") " Sep 08 00:45:29.426112 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:29.425981 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4s9n\" (UniqueName: \"kubernetes.io/projected/35bb5571-1ca6-46c2-9466-5caf9e15649f-kube-api-access-t4s9n\") pod \"35bb5571-1ca6-46c2-9466-5caf9e15649f\" (UID: \"35bb5571-1ca6-46c2-9466-5caf9e15649f\") " Sep 08 00:45:29.426112 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:29.426052 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bb5571-1ca6-46c2-9466-5caf9e15649f-plugins-config-volume\") pod \"35bb5571-1ca6-46c2-9466-5caf9e15649f\" (UID: \"35bb5571-1ca6-46c2-9466-5caf9e15649f\") " Sep 08 00:45:29.426112 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:29.426097 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0c77dd-42bc-43e3-bf19-3d34f3910617-plugins-config-volume\") pod \"9f0c77dd-42bc-43e3-bf19-3d34f3910617\" (UID: \"9f0c77dd-42bc-43e3-bf19-3d34f3910617\") " Sep 08 00:45:29.426466 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:29.426439 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35bb5571-1ca6-46c2-9466-5caf9e15649f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "35bb5571-1ca6-46c2-9466-5caf9e15649f" (UID: "35bb5571-1ca6-46c2-9466-5caf9e15649f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:45:29.426466 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:29.426451 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0c77dd-42bc-43e3-bf19-3d34f3910617-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f0c77dd-42bc-43e3-bf19-3d34f3910617" (UID: "9f0c77dd-42bc-43e3-bf19-3d34f3910617"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:45:29.428113 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:29.428090 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35bb5571-1ca6-46c2-9466-5caf9e15649f-kube-api-access-t4s9n" (OuterVolumeSpecName: "kube-api-access-t4s9n") pod "35bb5571-1ca6-46c2-9466-5caf9e15649f" (UID: "35bb5571-1ca6-46c2-9466-5caf9e15649f"). InnerVolumeSpecName "kube-api-access-t4s9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:45:29.428182 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:29.428108 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0c77dd-42bc-43e3-bf19-3d34f3910617-kube-api-access-2chnj" (OuterVolumeSpecName: "kube-api-access-2chnj") pod "9f0c77dd-42bc-43e3-bf19-3d34f3910617" (UID: "9f0c77dd-42bc-43e3-bf19-3d34f3910617"). InnerVolumeSpecName "kube-api-access-2chnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:45:29.527694 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:29.527644 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2chnj\" (UniqueName: \"kubernetes.io/projected/9f0c77dd-42bc-43e3-bf19-3d34f3910617-kube-api-access-2chnj\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:45:29.527694 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:29.527689 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4s9n\" (UniqueName: \"kubernetes.io/projected/35bb5571-1ca6-46c2-9466-5caf9e15649f-kube-api-access-t4s9n\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:45:29.527694 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:29.527701 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35bb5571-1ca6-46c2-9466-5caf9e15649f-plugins-config-volume\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:45:29.527939 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:29.527710 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0c77dd-42bc-43e3-bf19-3d34f3910617-plugins-config-volume\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:45:30.067325 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:30.067276 2577 generic.go:356] "Generic (PLEG): container finished" podID="9f0c77dd-42bc-43e3-bf19-3d34f3910617" containerID="5d79732a70873ecff63028738d9bd8018d50c5f497f3ee5540bdb450228a3a24" exitCode=0 Sep 08 00:45:30.067789 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:30.067327 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz" Sep 08 00:45:30.067789 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:30.067359 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz" event={"ID":"9f0c77dd-42bc-43e3-bf19-3d34f3910617","Type":"ContainerDied","Data":"5d79732a70873ecff63028738d9bd8018d50c5f497f3ee5540bdb450228a3a24"} Sep 08 00:45:30.067789 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:30.067426 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz" event={"ID":"9f0c77dd-42bc-43e3-bf19-3d34f3910617","Type":"ContainerDied","Data":"35fbb3a46f2b7ae3d2146391e6e3baedee84e86b59a6eef45af4de1d965f5d57"} Sep 08 00:45:30.067789 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:30.067459 2577 scope.go:117] "RemoveContainer" containerID="5d79732a70873ecff63028738d9bd8018d50c5f497f3ee5540bdb450228a3a24" Sep 08 00:45:30.069063 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:30.069038 2577 generic.go:356] "Generic (PLEG): container finished" podID="35bb5571-1ca6-46c2-9466-5caf9e15649f" containerID="876b54f8f6ebec1e959862666506e497bf787fea7c826edc1084870730213a91" exitCode=0 Sep 08 00:45:30.069167 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:30.069091 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm" Sep 08 00:45:30.069167 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:30.069113 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm" event={"ID":"35bb5571-1ca6-46c2-9466-5caf9e15649f","Type":"ContainerDied","Data":"876b54f8f6ebec1e959862666506e497bf787fea7c826edc1084870730213a91"} Sep 08 00:45:30.069167 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:30.069164 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm" event={"ID":"35bb5571-1ca6-46c2-9466-5caf9e15649f","Type":"ContainerDied","Data":"dfdd1948452bfb95c3737127d6ac0edc2fe4ba6685428ac2899c56e713d5866a"} Sep 08 00:45:30.077991 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:30.077964 2577 scope.go:117] "RemoveContainer" containerID="5d79732a70873ecff63028738d9bd8018d50c5f497f3ee5540bdb450228a3a24" Sep 08 00:45:30.078343 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:45:30.078320 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d79732a70873ecff63028738d9bd8018d50c5f497f3ee5540bdb450228a3a24\": container with ID starting with 5d79732a70873ecff63028738d9bd8018d50c5f497f3ee5540bdb450228a3a24 not found: ID does not exist" containerID="5d79732a70873ecff63028738d9bd8018d50c5f497f3ee5540bdb450228a3a24" Sep 08 00:45:30.078436 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:30.078353 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d79732a70873ecff63028738d9bd8018d50c5f497f3ee5540bdb450228a3a24"} err="failed to get container status \"5d79732a70873ecff63028738d9bd8018d50c5f497f3ee5540bdb450228a3a24\": rpc error: code = NotFound desc = could not find container \"5d79732a70873ecff63028738d9bd8018d50c5f497f3ee5540bdb450228a3a24\": container with ID starting with 5d79732a70873ecff63028738d9bd8018d50c5f497f3ee5540bdb450228a3a24 not found: ID does not exist" Sep 08 00:45:30.078436 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:30.078376 2577 scope.go:117] "RemoveContainer" containerID="876b54f8f6ebec1e959862666506e497bf787fea7c826edc1084870730213a91" Sep 08 00:45:30.086163 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:30.086144 2577 scope.go:117] "RemoveContainer" containerID="876b54f8f6ebec1e959862666506e497bf787fea7c826edc1084870730213a91" Sep 08 00:45:30.086411 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:45:30.086391 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"876b54f8f6ebec1e959862666506e497bf787fea7c826edc1084870730213a91\": container with ID starting with 876b54f8f6ebec1e959862666506e497bf787fea7c826edc1084870730213a91 not found: ID does not exist" containerID="876b54f8f6ebec1e959862666506e497bf787fea7c826edc1084870730213a91" Sep 08 00:45:30.086471 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:30.086417 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"876b54f8f6ebec1e959862666506e497bf787fea7c826edc1084870730213a91"} err="failed to get container status \"876b54f8f6ebec1e959862666506e497bf787fea7c826edc1084870730213a91\": rpc error: code = NotFound desc = could not find container \"876b54f8f6ebec1e959862666506e497bf787fea7c826edc1084870730213a91\": container with ID starting with 876b54f8f6ebec1e959862666506e497bf787fea7c826edc1084870730213a91 not found: ID does not exist" Sep 08 00:45:30.094905 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:30.094886 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm"] Sep 08 00:45:30.098313 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:30.098291 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f80520f6-d8f4c9894-nmfxm"] Sep 08 00:45:30.112420 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:30.112397 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz"] Sep 08 00:45:30.117978 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:30.117957 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f80520f6-799fdcdf45-9gbbz"] Sep 08 00:45:30.791198 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:30.791163 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35bb5571-1ca6-46c2-9466-5caf9e15649f" path="/var/lib/kubelet/pods/35bb5571-1ca6-46c2-9466-5caf9e15649f/volumes" Sep 08 00:45:30.791578 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:30.791558 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eb84fcc-a190-4210-9407-cb481344f93c" path="/var/lib/kubelet/pods/4eb84fcc-a190-4210-9407-cb481344f93c/volumes" Sep 08 00:45:30.791940 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:30.791922 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f0c77dd-42bc-43e3-bf19-3d34f3910617" path="/var/lib/kubelet/pods/9f0c77dd-42bc-43e3-bf19-3d34f3910617/volumes" Sep 08 00:45:35.546506 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.546471 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd"] Sep 08 00:45:35.546891 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.546788 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eb84fcc-a190-4210-9407-cb481344f93c" containerName="revoke-keys" Sep 08 00:45:35.546891 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.546798 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eb84fcc-a190-4210-9407-cb481344f93c" containerName="revoke-keys" Sep 08 00:45:35.546891 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.546810 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35bb5571-1ca6-46c2-9466-5caf9e15649f" containerName="payload-pre-processing" Sep 08 00:45:35.546891 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.546815 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="35bb5571-1ca6-46c2-9466-5caf9e15649f" containerName="payload-pre-processing" Sep 08 00:45:35.546891 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.546829 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f0c77dd-42bc-43e3-bf19-3d34f3910617" containerName="payload-processing" Sep 08 00:45:35.546891 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.546835 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f0c77dd-42bc-43e3-bf19-3d34f3910617" containerName="payload-processing" Sep 08 00:45:35.546891 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.546882 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="35bb5571-1ca6-46c2-9466-5caf9e15649f" containerName="payload-pre-processing" Sep 08 00:45:35.546891 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.546891 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eb84fcc-a190-4210-9407-cb481344f93c" containerName="revoke-keys" Sep 08 00:45:35.547218 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.546899 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f0c77dd-42bc-43e3-bf19-3d34f3910617" containerName="payload-processing" Sep 08 00:45:35.550021 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.549983 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd" Sep 08 00:45:35.552670 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.552645 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-9b001d57-dockercfg-6lb62\"" Sep 08 00:45:35.552786 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.552668 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-9b001d57\"" Sep 08 00:45:35.558241 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.558218 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd"] Sep 08 00:45:35.570326 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.570300 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp"] Sep 08 00:45:35.573570 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.573550 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp" Sep 08 00:45:35.589730 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.589703 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp"] Sep 08 00:45:35.682337 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.682279 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e86202e9-fe00-4974-8a68-70a551f24d84-plugins-config-volume\") pod \"payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp\" (UID: \"e86202e9-fe00-4974-8a68-70a551f24d84\") " pod="openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp" Sep 08 00:45:35.682527 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.682413 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056ec851-30aa-44eb-b6b6-ace8a734d328-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd\" (UID: \"056ec851-30aa-44eb-b6b6-ace8a734d328\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd" Sep 08 00:45:35.682527 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.682451 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjdtz\" (UniqueName: \"kubernetes.io/projected/e86202e9-fe00-4974-8a68-70a551f24d84-kube-api-access-kjdtz\") pod \"payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp\" (UID: \"e86202e9-fe00-4974-8a68-70a551f24d84\") " pod="openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp" Sep 08 00:45:35.682527 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.682499 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4vj8\" (UniqueName: \"kubernetes.io/projected/056ec851-30aa-44eb-b6b6-ace8a734d328-kube-api-access-n4vj8\") pod \"payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd\" (UID: \"056ec851-30aa-44eb-b6b6-ace8a734d328\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd" Sep 08 00:45:35.783284 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.783247 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056ec851-30aa-44eb-b6b6-ace8a734d328-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd\" (UID: \"056ec851-30aa-44eb-b6b6-ace8a734d328\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd" Sep 08 00:45:35.783284 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.783287 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjdtz\" (UniqueName: \"kubernetes.io/projected/e86202e9-fe00-4974-8a68-70a551f24d84-kube-api-access-kjdtz\") pod \"payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp\" (UID: \"e86202e9-fe00-4974-8a68-70a551f24d84\") " pod="openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp" Sep 08 00:45:35.783492 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.783326 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4vj8\" (UniqueName: \"kubernetes.io/projected/056ec851-30aa-44eb-b6b6-ace8a734d328-kube-api-access-n4vj8\") pod \"payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd\" (UID: \"056ec851-30aa-44eb-b6b6-ace8a734d328\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd" Sep 08 00:45:35.783492 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.783372 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e86202e9-fe00-4974-8a68-70a551f24d84-plugins-config-volume\") pod \"payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp\" (UID: \"e86202e9-fe00-4974-8a68-70a551f24d84\") " pod="openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp" Sep 08 00:45:35.784033 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.783982 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056ec851-30aa-44eb-b6b6-ace8a734d328-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd\" (UID: \"056ec851-30aa-44eb-b6b6-ace8a734d328\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd" Sep 08 00:45:35.784137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.784119 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e86202e9-fe00-4974-8a68-70a551f24d84-plugins-config-volume\") pod \"payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp\" (UID: \"e86202e9-fe00-4974-8a68-70a551f24d84\") " pod="openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp" Sep 08 00:45:35.791390 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.791364 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4vj8\" (UniqueName: \"kubernetes.io/projected/056ec851-30aa-44eb-b6b6-ace8a734d328-kube-api-access-n4vj8\") pod \"payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd\" (UID: \"056ec851-30aa-44eb-b6b6-ace8a734d328\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd" Sep 08 00:45:35.791496 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.791430 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjdtz\" (UniqueName: \"kubernetes.io/projected/e86202e9-fe00-4974-8a68-70a551f24d84-kube-api-access-kjdtz\") pod \"payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp\" (UID: \"e86202e9-fe00-4974-8a68-70a551f24d84\") " pod="openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp" Sep 08 00:45:35.861341 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.861271 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd" Sep 08 00:45:35.883451 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:35.883424 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp" Sep 08 00:45:36.133464 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:36.133440 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd"] Sep 08 00:45:36.134474 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:45:36.134450 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056ec851_30aa_44eb_b6b6_ace8a734d328.slice/crio-2ec2bb7d459601d6fe9fe97e7ccf971fd19a425939ca5fe0a358a9aa31154e15 WatchSource:0}: Error finding container 2ec2bb7d459601d6fe9fe97e7ccf971fd19a425939ca5fe0a358a9aa31154e15: Status 404 returned error can't find the container with id 2ec2bb7d459601d6fe9fe97e7ccf971fd19a425939ca5fe0a358a9aa31154e15 Sep 08 00:45:36.163902 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:36.163882 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp"] Sep 08 00:45:36.167015 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:45:36.166972 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86202e9_fe00_4974_8a68_70a551f24d84.slice/crio-8e0c7bc610dc08bd80990ce74f3f8c07ded7502afb54075a2201402d756aab1a WatchSource:0}: Error finding container 8e0c7bc610dc08bd80990ce74f3f8c07ded7502afb54075a2201402d756aab1a: Status 404 returned error can't find the container with id 8e0c7bc610dc08bd80990ce74f3f8c07ded7502afb54075a2201402d756aab1a Sep 08 00:45:37.106195 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:37.106153 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd" event={"ID":"056ec851-30aa-44eb-b6b6-ace8a734d328","Type":"ContainerStarted","Data":"fd47e37a8170fb9e442566c7ee9ee44739e4f88083bed37eaf3fe6eb34c361dc"} Sep 08 00:45:37.106661 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:37.106216 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd" event={"ID":"056ec851-30aa-44eb-b6b6-ace8a734d328","Type":"ContainerStarted","Data":"2ec2bb7d459601d6fe9fe97e7ccf971fd19a425939ca5fe0a358a9aa31154e15"} Sep 08 00:45:37.106661 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:37.106323 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd" Sep 08 00:45:37.107729 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:37.107707 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp" event={"ID":"e86202e9-fe00-4974-8a68-70a551f24d84","Type":"ContainerStarted","Data":"d44afa26dbffb55c1d1003f1b8b9f6dec32500344b60d0544be1d5846f0d9793"} Sep 08 00:45:37.107852 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:37.107747 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp" event={"ID":"e86202e9-fe00-4974-8a68-70a551f24d84","Type":"ContainerStarted","Data":"8e0c7bc610dc08bd80990ce74f3f8c07ded7502afb54075a2201402d756aab1a"} Sep 08 00:45:37.107852 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:37.107784 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp" Sep 08 00:45:37.124184 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:37.124131 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd" podStartSLOduration=2.124114003 podStartE2EDuration="2.124114003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:45:37.122294348 +0000 UTC m=+750.817535509" watchObservedRunningTime="2026-09-08 00:45:37.124114003 +0000 UTC m=+750.819355162" Sep 08 00:45:37.139965 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:37.139912 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp" podStartSLOduration=2.13989525 podStartE2EDuration="2.13989525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:45:37.137674715 +0000 UTC m=+750.832915875" watchObservedRunningTime="2026-09-08 00:45:37.13989525 +0000 UTC m=+750.835136408" Sep 08 00:45:48.120845 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:48.120810 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd" Sep 08 00:45:48.121296 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:48.120980 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp" Sep 08 00:45:51.478246 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:51.478208 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-31f424-rp8c7"] Sep 08 00:45:51.481620 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:51.481602 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-31f424-rp8c7" Sep 08 00:45:51.484123 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:51.484101 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 00:45:51.484235 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:51.484213 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dqw2\"" Sep 08 00:45:51.489528 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:51.489185 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-31f424-rp8c7"] Sep 08 00:45:51.513292 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:51.513256 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77092686-5475-4bdc-a719-4b8b039d167c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-31f424-rp8c7\" (UID: \"77092686-5475-4bdc-a719-4b8b039d167c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-31f424-rp8c7" Sep 08 00:45:51.614497 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:51.614466 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77092686-5475-4bdc-a719-4b8b039d167c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-31f424-rp8c7\" (UID: \"77092686-5475-4bdc-a719-4b8b039d167c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-31f424-rp8c7" Sep 08 00:45:51.615047 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:51.615027 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77092686-5475-4bdc-a719-4b8b039d167c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-31f424-rp8c7\" (UID: \"77092686-5475-4bdc-a719-4b8b039d167c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-31f424-rp8c7" Sep 08 00:45:51.794349 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:51.794317 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-31f424-rp8c7" Sep 08 00:45:52.068438 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:52.068410 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-31f424-rp8c7"] Sep 08 00:45:52.070502 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:45:52.070476 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77092686_5475_4bdc_a719_4b8b039d167c.slice/crio-01ab59e3e7d7ee11abf949640d55ff38a405547fd2aed5fb3bb67c4beeaf54db WatchSource:0}: Error finding container 01ab59e3e7d7ee11abf949640d55ff38a405547fd2aed5fb3bb67c4beeaf54db: Status 404 returned error can't find the container with id 01ab59e3e7d7ee11abf949640d55ff38a405547fd2aed5fb3bb67c4beeaf54db Sep 08 00:45:52.188783 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:52.188748 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-31f424-rp8c7" event={"ID":"77092686-5475-4bdc-a719-4b8b039d167c","Type":"ContainerStarted","Data":"01ab59e3e7d7ee11abf949640d55ff38a405547fd2aed5fb3bb67c4beeaf54db"} Sep 08 00:45:53.195237 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:53.195142 2577 generic.go:356] "Generic (PLEG): container finished" podID="77092686-5475-4bdc-a719-4b8b039d167c" containerID="06648f903a1d3e5cd9bf5d87f84c7dddc324b430bf18276b27fa49ee0be77779" exitCode=0 Sep 08 00:45:53.195627 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:53.195230 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-31f424-rp8c7" event={"ID":"77092686-5475-4bdc-a719-4b8b039d167c","Type":"ContainerDied","Data":"06648f903a1d3e5cd9bf5d87f84c7dddc324b430bf18276b27fa49ee0be77779"} Sep 08 00:45:54.331910 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:54.331886 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-31f424-rp8c7" Sep 08 00:45:54.436199 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:54.436166 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77092686-5475-4bdc-a719-4b8b039d167c-maas-api-service-ca\") pod \"77092686-5475-4bdc-a719-4b8b039d167c\" (UID: \"77092686-5475-4bdc-a719-4b8b039d167c\") " Sep 08 00:45:54.436521 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:54.436498 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77092686-5475-4bdc-a719-4b8b039d167c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "77092686-5475-4bdc-a719-4b8b039d167c" (UID: "77092686-5475-4bdc-a719-4b8b039d167c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:45:54.537395 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:54.537369 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77092686-5475-4bdc-a719-4b8b039d167c-maas-api-service-ca\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:45:55.208690 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:55.208604 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-31f424-rp8c7" Sep 08 00:45:55.208844 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:55.208602 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-31f424-rp8c7" event={"ID":"77092686-5475-4bdc-a719-4b8b039d167c","Type":"ContainerDied","Data":"01ab59e3e7d7ee11abf949640d55ff38a405547fd2aed5fb3bb67c4beeaf54db"} Sep 08 00:45:55.208844 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:55.208746 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01ab59e3e7d7ee11abf949640d55ff38a405547fd2aed5fb3bb67c4beeaf54db" Sep 08 00:45:58.337670 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:58.337635 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b001d57-5mqk2"] Sep 08 00:45:58.338134 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:58.337986 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77092686-5475-4bdc-a719-4b8b039d167c" containerName="revoke-keys" Sep 08 00:45:58.338134 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:58.338011 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="77092686-5475-4bdc-a719-4b8b039d167c" containerName="revoke-keys" Sep 08 00:45:58.338134 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:58.338088 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="77092686-5475-4bdc-a719-4b8b039d167c" containerName="revoke-keys" Sep 08 00:45:58.340968 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:58.340951 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b001d57-5mqk2" Sep 08 00:45:58.343372 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:58.343351 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dqw2\"" Sep 08 00:45:58.343504 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:58.343356 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 00:45:58.347161 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:58.347138 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b001d57-5mqk2"] Sep 08 00:45:58.471375 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:58.471339 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9b001d57-5mqk2\" (UID: \"e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b001d57-5mqk2" Sep 08 00:45:58.572700 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:58.572669 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9b001d57-5mqk2\" (UID: \"e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b001d57-5mqk2" Sep 08 00:45:58.573278 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:58.573257 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9b001d57-5mqk2\" (UID: \"e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b001d57-5mqk2" Sep 08 00:45:58.652408 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:58.652329 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b001d57-5mqk2" Sep 08 00:45:58.923448 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:58.923423 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b001d57-5mqk2"] Sep 08 00:45:58.925436 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:45:58.925413 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f96c56_8c6c_4a27_b6c3_35ade0ae16ec.slice/crio-7d214f6ab8556cf198b2b3d5bfd2d1300f29f43d61704bc90e0b3ae600200b35 WatchSource:0}: Error finding container 7d214f6ab8556cf198b2b3d5bfd2d1300f29f43d61704bc90e0b3ae600200b35: Status 404 returned error can't find the container with id 7d214f6ab8556cf198b2b3d5bfd2d1300f29f43d61704bc90e0b3ae600200b35 Sep 08 00:45:59.231043 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:59.231008 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b001d57-5mqk2" event={"ID":"e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec","Type":"ContainerStarted","Data":"2c1a416bba4c63340074c29ffb33047e11ae95be56a9154aba00c12c058fea12"} Sep 08 00:45:59.231166 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:59.231067 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b001d57-5mqk2" event={"ID":"e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec","Type":"ContainerStarted","Data":"7d214f6ab8556cf198b2b3d5bfd2d1300f29f43d61704bc90e0b3ae600200b35"} Sep 08 00:45:59.245348 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:45:59.245301 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b001d57-5mqk2" podStartSLOduration=1.24528549 podStartE2EDuration="1.24528549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:45:59.244189445 +0000 UTC m=+772.939430605" watchObservedRunningTime="2026-09-08 00:45:59.24528549 +0000 UTC m=+772.940526649" Sep 08 00:46:00.237387 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:00.237348 2577 generic.go:356] "Generic (PLEG): container finished" podID="e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec" containerID="2c1a416bba4c63340074c29ffb33047e11ae95be56a9154aba00c12c058fea12" exitCode=0 Sep 08 00:46:00.237797 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:00.237439 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b001d57-5mqk2" event={"ID":"e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec","Type":"ContainerDied","Data":"2c1a416bba4c63340074c29ffb33047e11ae95be56a9154aba00c12c058fea12"} Sep 08 00:46:01.369404 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:01.369374 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b001d57-5mqk2" Sep 08 00:46:01.499676 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:01.499582 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec-maas-api-service-ca\") pod \"e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec\" (UID: \"e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec\") " Sep 08 00:46:01.499969 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:01.499943 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec" (UID: "e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:46:01.600375 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:01.600342 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec-maas-api-service-ca\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:46:02.203077 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.203043 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-31f424-29813805-vm89z"] Sep 08 00:46:02.204637 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.204615 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-31f424-29813805-vm89z"] Sep 08 00:46:02.237210 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.237178 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z"] Sep 08 00:46:02.237483 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.237453 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z" podUID="80f84a3d-f6fb-4067-bf08-629980db649b" containerName="payload-processing" containerID="cri-o://6d7f1c4dfc50a873b46d7a0d81ef718580fe45c7032578c9b1dd1322dbd52bfe" gracePeriod=30 Sep 08 00:46:02.246283 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.246248 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl"] Sep 08 00:46:02.246671 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.246500 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl" podUID="f0654cf5-8dd6-4180-b7f1-5b83cfc598b2" containerName="payload-pre-processing" containerID="cri-o://b1a9004f2e51277206d946aaa1a07a74ba0da2026edeb34aab742040e55e9d4f" gracePeriod=30 Sep 08 00:46:02.253350 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.253250 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b001d57-5mqk2" event={"ID":"e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec","Type":"ContainerDied","Data":"7d214f6ab8556cf198b2b3d5bfd2d1300f29f43d61704bc90e0b3ae600200b35"} Sep 08 00:46:02.253485 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.253329 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d214f6ab8556cf198b2b3d5bfd2d1300f29f43d61704bc90e0b3ae600200b35" Sep 08 00:46:02.253605 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.253540 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b001d57-5mqk2" Sep 08 00:46:02.405163 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.405120 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-31f424-rp8c7"] Sep 08 00:46:02.412356 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.412309 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-31f424-rp8c7"] Sep 08 00:46:02.490520 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.490492 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z" Sep 08 00:46:02.533479 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.533454 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl" Sep 08 00:46:02.609209 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.609180 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7ckl\" (UniqueName: \"kubernetes.io/projected/f0654cf5-8dd6-4180-b7f1-5b83cfc598b2-kube-api-access-b7ckl\") pod \"f0654cf5-8dd6-4180-b7f1-5b83cfc598b2\" (UID: \"f0654cf5-8dd6-4180-b7f1-5b83cfc598b2\") " Sep 08 00:46:02.609368 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.609219 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cl4cl\" (UniqueName: \"kubernetes.io/projected/80f84a3d-f6fb-4067-bf08-629980db649b-kube-api-access-cl4cl\") pod \"80f84a3d-f6fb-4067-bf08-629980db649b\" (UID: \"80f84a3d-f6fb-4067-bf08-629980db649b\") " Sep 08 00:46:02.609368 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.609264 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f84a3d-f6fb-4067-bf08-629980db649b-plugins-config-volume\") pod \"80f84a3d-f6fb-4067-bf08-629980db649b\" (UID: \"80f84a3d-f6fb-4067-bf08-629980db649b\") " Sep 08 00:46:02.609368 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.609302 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0654cf5-8dd6-4180-b7f1-5b83cfc598b2-plugins-config-volume\") pod \"f0654cf5-8dd6-4180-b7f1-5b83cfc598b2\" (UID: \"f0654cf5-8dd6-4180-b7f1-5b83cfc598b2\") " Sep 08 00:46:02.609688 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.609656 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f84a3d-f6fb-4067-bf08-629980db649b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "80f84a3d-f6fb-4067-bf08-629980db649b" (UID: "80f84a3d-f6fb-4067-bf08-629980db649b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:46:02.609800 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.609717 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0654cf5-8dd6-4180-b7f1-5b83cfc598b2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f0654cf5-8dd6-4180-b7f1-5b83cfc598b2" (UID: "f0654cf5-8dd6-4180-b7f1-5b83cfc598b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:46:02.611365 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.611339 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f84a3d-f6fb-4067-bf08-629980db649b-kube-api-access-cl4cl" (OuterVolumeSpecName: "kube-api-access-cl4cl") pod "80f84a3d-f6fb-4067-bf08-629980db649b" (UID: "80f84a3d-f6fb-4067-bf08-629980db649b"). InnerVolumeSpecName "kube-api-access-cl4cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:46:02.611444 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.611401 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0654cf5-8dd6-4180-b7f1-5b83cfc598b2-kube-api-access-b7ckl" (OuterVolumeSpecName: "kube-api-access-b7ckl") pod "f0654cf5-8dd6-4180-b7f1-5b83cfc598b2" (UID: "f0654cf5-8dd6-4180-b7f1-5b83cfc598b2"). InnerVolumeSpecName "kube-api-access-b7ckl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:46:02.710346 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.710268 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7ckl\" (UniqueName: \"kubernetes.io/projected/f0654cf5-8dd6-4180-b7f1-5b83cfc598b2-kube-api-access-b7ckl\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:46:02.710346 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.710293 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl4cl\" (UniqueName: \"kubernetes.io/projected/80f84a3d-f6fb-4067-bf08-629980db649b-kube-api-access-cl4cl\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:46:02.710346 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.710302 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80f84a3d-f6fb-4067-bf08-629980db649b-plugins-config-volume\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:46:02.710346 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.710310 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0654cf5-8dd6-4180-b7f1-5b83cfc598b2-plugins-config-volume\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:46:02.790425 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.790392 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d439106-d1eb-43a8-8da0-ac9e7c47c086" path="/var/lib/kubelet/pods/3d439106-d1eb-43a8-8da0-ac9e7c47c086/volumes" Sep 08 00:46:02.790698 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:02.790685 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77092686-5475-4bdc-a719-4b8b039d167c" path="/var/lib/kubelet/pods/77092686-5475-4bdc-a719-4b8b039d167c/volumes" Sep 08 00:46:03.259773 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:03.259732 2577 generic.go:356] "Generic (PLEG): container finished" podID="f0654cf5-8dd6-4180-b7f1-5b83cfc598b2" containerID="b1a9004f2e51277206d946aaa1a07a74ba0da2026edeb34aab742040e55e9d4f" exitCode=0 Sep 08 00:46:03.259773 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:03.259776 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl" Sep 08 00:46:03.259970 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:03.259821 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl" event={"ID":"f0654cf5-8dd6-4180-b7f1-5b83cfc598b2","Type":"ContainerDied","Data":"b1a9004f2e51277206d946aaa1a07a74ba0da2026edeb34aab742040e55e9d4f"} Sep 08 00:46:03.259970 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:03.259885 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl" event={"ID":"f0654cf5-8dd6-4180-b7f1-5b83cfc598b2","Type":"ContainerDied","Data":"78e9d14d35dcb37504731a5856ae9917b13727a6f291c37f723f83b2e8e4fd07"} Sep 08 00:46:03.259970 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:03.259921 2577 scope.go:117] "RemoveContainer" containerID="b1a9004f2e51277206d946aaa1a07a74ba0da2026edeb34aab742040e55e9d4f" Sep 08 00:46:03.261366 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:03.261241 2577 generic.go:356] "Generic (PLEG): container finished" podID="80f84a3d-f6fb-4067-bf08-629980db649b" containerID="6d7f1c4dfc50a873b46d7a0d81ef718580fe45c7032578c9b1dd1322dbd52bfe" exitCode=0 Sep 08 00:46:03.261366 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:03.261277 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z" Sep 08 00:46:03.261366 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:03.261310 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z" event={"ID":"80f84a3d-f6fb-4067-bf08-629980db649b","Type":"ContainerDied","Data":"6d7f1c4dfc50a873b46d7a0d81ef718580fe45c7032578c9b1dd1322dbd52bfe"} Sep 08 00:46:03.261366 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:03.261349 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z" event={"ID":"80f84a3d-f6fb-4067-bf08-629980db649b","Type":"ContainerDied","Data":"d63a92c3ba724a85b60819cec7dc1e0d8fb09dcfdfa77c1da58e6fc4861fc087"} Sep 08 00:46:03.268798 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:03.268782 2577 scope.go:117] "RemoveContainer" containerID="b1a9004f2e51277206d946aaa1a07a74ba0da2026edeb34aab742040e55e9d4f" Sep 08 00:46:03.269081 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:46:03.269060 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a9004f2e51277206d946aaa1a07a74ba0da2026edeb34aab742040e55e9d4f\": container with ID starting with b1a9004f2e51277206d946aaa1a07a74ba0da2026edeb34aab742040e55e9d4f not found: ID does not exist" containerID="b1a9004f2e51277206d946aaa1a07a74ba0da2026edeb34aab742040e55e9d4f" Sep 08 00:46:03.269131 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:03.269090 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a9004f2e51277206d946aaa1a07a74ba0da2026edeb34aab742040e55e9d4f"} err="failed to get container status \"b1a9004f2e51277206d946aaa1a07a74ba0da2026edeb34aab742040e55e9d4f\": rpc error: code = NotFound desc = could not find container \"b1a9004f2e51277206d946aaa1a07a74ba0da2026edeb34aab742040e55e9d4f\": container with ID starting with b1a9004f2e51277206d946aaa1a07a74ba0da2026edeb34aab742040e55e9d4f not found: ID does not exist" Sep 08 00:46:03.269131 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:03.269105 2577 scope.go:117] "RemoveContainer" containerID="6d7f1c4dfc50a873b46d7a0d81ef718580fe45c7032578c9b1dd1322dbd52bfe" Sep 08 00:46:03.282794 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:03.277030 2577 scope.go:117] "RemoveContainer" containerID="6d7f1c4dfc50a873b46d7a0d81ef718580fe45c7032578c9b1dd1322dbd52bfe" Sep 08 00:46:03.282794 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:46:03.277342 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d7f1c4dfc50a873b46d7a0d81ef718580fe45c7032578c9b1dd1322dbd52bfe\": container with ID starting with 6d7f1c4dfc50a873b46d7a0d81ef718580fe45c7032578c9b1dd1322dbd52bfe not found: ID does not exist" containerID="6d7f1c4dfc50a873b46d7a0d81ef718580fe45c7032578c9b1dd1322dbd52bfe" Sep 08 00:46:03.282794 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:03.277373 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d7f1c4dfc50a873b46d7a0d81ef718580fe45c7032578c9b1dd1322dbd52bfe"} err="failed to get container status \"6d7f1c4dfc50a873b46d7a0d81ef718580fe45c7032578c9b1dd1322dbd52bfe\": rpc error: code = NotFound desc = could not find container \"6d7f1c4dfc50a873b46d7a0d81ef718580fe45c7032578c9b1dd1322dbd52bfe\": container with ID starting with 6d7f1c4dfc50a873b46d7a0d81ef718580fe45c7032578c9b1dd1322dbd52bfe not found: ID does not exist" Sep 08 00:46:03.283337 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:03.283317 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl"] Sep 08 00:46:03.285081 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:03.285060 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-31f424-84bcd66fcd-fjtdl"] Sep 08 00:46:03.293038 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:03.293020 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z"] Sep 08 00:46:03.296915 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:03.296896 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-31f424-64474cfdbf-m7f9z"] Sep 08 00:46:04.791184 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:04.791151 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80f84a3d-f6fb-4067-bf08-629980db649b" path="/var/lib/kubelet/pods/80f84a3d-f6fb-4067-bf08-629980db649b/volumes" Sep 08 00:46:04.791557 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:04.791496 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0654cf5-8dd6-4180-b7f1-5b83cfc598b2" path="/var/lib/kubelet/pods/f0654cf5-8dd6-4180-b7f1-5b83cfc598b2/volumes" Sep 08 00:46:05.553647 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:05.553612 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp"] Sep 08 00:46:05.553931 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:05.553896 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp" podUID="e86202e9-fe00-4974-8a68-70a551f24d84" containerName="payload-processing" containerID="cri-o://d44afa26dbffb55c1d1003f1b8b9f6dec32500344b60d0544be1d5846f0d9793" gracePeriod=30 Sep 08 00:46:05.561277 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:05.561248 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd"] Sep 08 00:46:05.561498 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:05.561475 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd" podUID="056ec851-30aa-44eb-b6b6-ace8a734d328" containerName="payload-pre-processing" containerID="cri-o://fd47e37a8170fb9e442566c7ee9ee44739e4f88083bed37eaf3fe6eb34c361dc" gracePeriod=30 Sep 08 00:46:05.714512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:05.714419 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b001d57-5mqk2"] Sep 08 00:46:05.716478 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:05.716445 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9b001d57-5mqk2"] Sep 08 00:46:05.831448 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:05.831328 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp" Sep 08 00:46:05.835225 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:05.835202 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd" Sep 08 00:46:05.936489 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:05.936457 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4vj8\" (UniqueName: \"kubernetes.io/projected/056ec851-30aa-44eb-b6b6-ace8a734d328-kube-api-access-n4vj8\") pod \"056ec851-30aa-44eb-b6b6-ace8a734d328\" (UID: \"056ec851-30aa-44eb-b6b6-ace8a734d328\") " Sep 08 00:46:05.936672 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:05.936524 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e86202e9-fe00-4974-8a68-70a551f24d84-plugins-config-volume\") pod \"e86202e9-fe00-4974-8a68-70a551f24d84\" (UID: \"e86202e9-fe00-4974-8a68-70a551f24d84\") " Sep 08 00:46:05.936672 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:05.936566 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056ec851-30aa-44eb-b6b6-ace8a734d328-plugins-config-volume\") pod \"056ec851-30aa-44eb-b6b6-ace8a734d328\" (UID: \"056ec851-30aa-44eb-b6b6-ace8a734d328\") " Sep 08 00:46:05.936672 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:05.936592 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjdtz\" (UniqueName: \"kubernetes.io/projected/e86202e9-fe00-4974-8a68-70a551f24d84-kube-api-access-kjdtz\") pod \"e86202e9-fe00-4974-8a68-70a551f24d84\" (UID: \"e86202e9-fe00-4974-8a68-70a551f24d84\") " Sep 08 00:46:05.936920 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:05.936895 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e86202e9-fe00-4974-8a68-70a551f24d84-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e86202e9-fe00-4974-8a68-70a551f24d84" (UID: "e86202e9-fe00-4974-8a68-70a551f24d84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:46:05.936981 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:05.936931 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056ec851-30aa-44eb-b6b6-ace8a734d328-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "056ec851-30aa-44eb-b6b6-ace8a734d328" (UID: "056ec851-30aa-44eb-b6b6-ace8a734d328"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:46:05.938705 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:05.938680 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056ec851-30aa-44eb-b6b6-ace8a734d328-kube-api-access-n4vj8" (OuterVolumeSpecName: "kube-api-access-n4vj8") pod "056ec851-30aa-44eb-b6b6-ace8a734d328" (UID: "056ec851-30aa-44eb-b6b6-ace8a734d328"). InnerVolumeSpecName "kube-api-access-n4vj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:46:05.938810 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:05.938720 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86202e9-fe00-4974-8a68-70a551f24d84-kube-api-access-kjdtz" (OuterVolumeSpecName: "kube-api-access-kjdtz") pod "e86202e9-fe00-4974-8a68-70a551f24d84" (UID: "e86202e9-fe00-4974-8a68-70a551f24d84"). InnerVolumeSpecName "kube-api-access-kjdtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:46:06.037555 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.037522 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e86202e9-fe00-4974-8a68-70a551f24d84-plugins-config-volume\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:46:06.037555 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.037547 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056ec851-30aa-44eb-b6b6-ace8a734d328-plugins-config-volume\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:46:06.037555 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.037557 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjdtz\" (UniqueName: \"kubernetes.io/projected/e86202e9-fe00-4974-8a68-70a551f24d84-kube-api-access-kjdtz\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:46:06.037555 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.037565 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4vj8\" (UniqueName: \"kubernetes.io/projected/056ec851-30aa-44eb-b6b6-ace8a734d328-kube-api-access-n4vj8\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:46:06.279949 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.279920 2577 generic.go:356] "Generic (PLEG): container finished" podID="e86202e9-fe00-4974-8a68-70a551f24d84" containerID="d44afa26dbffb55c1d1003f1b8b9f6dec32500344b60d0544be1d5846f0d9793" exitCode=0 Sep 08 00:46:06.280143 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.279973 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp" Sep 08 00:46:06.280143 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.280019 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp" event={"ID":"e86202e9-fe00-4974-8a68-70a551f24d84","Type":"ContainerDied","Data":"d44afa26dbffb55c1d1003f1b8b9f6dec32500344b60d0544be1d5846f0d9793"} Sep 08 00:46:06.280143 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.280070 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp" event={"ID":"e86202e9-fe00-4974-8a68-70a551f24d84","Type":"ContainerDied","Data":"8e0c7bc610dc08bd80990ce74f3f8c07ded7502afb54075a2201402d756aab1a"} Sep 08 00:46:06.280143 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.280100 2577 scope.go:117] "RemoveContainer" containerID="d44afa26dbffb55c1d1003f1b8b9f6dec32500344b60d0544be1d5846f0d9793" Sep 08 00:46:06.281392 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.281370 2577 generic.go:356] "Generic (PLEG): container finished" podID="056ec851-30aa-44eb-b6b6-ace8a734d328" containerID="fd47e37a8170fb9e442566c7ee9ee44739e4f88083bed37eaf3fe6eb34c361dc" exitCode=0 Sep 08 00:46:06.281509 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.281408 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd" event={"ID":"056ec851-30aa-44eb-b6b6-ace8a734d328","Type":"ContainerDied","Data":"fd47e37a8170fb9e442566c7ee9ee44739e4f88083bed37eaf3fe6eb34c361dc"} Sep 08 00:46:06.281509 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.281422 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd" Sep 08 00:46:06.281509 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.281453 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd" event={"ID":"056ec851-30aa-44eb-b6b6-ace8a734d328","Type":"ContainerDied","Data":"2ec2bb7d459601d6fe9fe97e7ccf971fd19a425939ca5fe0a358a9aa31154e15"} Sep 08 00:46:06.288838 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.288814 2577 scope.go:117] "RemoveContainer" containerID="d44afa26dbffb55c1d1003f1b8b9f6dec32500344b60d0544be1d5846f0d9793" Sep 08 00:46:06.289143 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:46:06.289122 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d44afa26dbffb55c1d1003f1b8b9f6dec32500344b60d0544be1d5846f0d9793\": container with ID starting with d44afa26dbffb55c1d1003f1b8b9f6dec32500344b60d0544be1d5846f0d9793 not found: ID does not exist" containerID="d44afa26dbffb55c1d1003f1b8b9f6dec32500344b60d0544be1d5846f0d9793" Sep 08 00:46:06.289217 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.289152 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d44afa26dbffb55c1d1003f1b8b9f6dec32500344b60d0544be1d5846f0d9793"} err="failed to get container status \"d44afa26dbffb55c1d1003f1b8b9f6dec32500344b60d0544be1d5846f0d9793\": rpc error: code = NotFound desc = could not find container \"d44afa26dbffb55c1d1003f1b8b9f6dec32500344b60d0544be1d5846f0d9793\": container with ID starting with d44afa26dbffb55c1d1003f1b8b9f6dec32500344b60d0544be1d5846f0d9793 not found: ID does not exist" Sep 08 00:46:06.289217 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.289169 2577 scope.go:117] "RemoveContainer" containerID="fd47e37a8170fb9e442566c7ee9ee44739e4f88083bed37eaf3fe6eb34c361dc" Sep 08 00:46:06.297472 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.297458 2577 scope.go:117] "RemoveContainer" containerID="fd47e37a8170fb9e442566c7ee9ee44739e4f88083bed37eaf3fe6eb34c361dc" Sep 08 00:46:06.297742 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:46:06.297725 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd47e37a8170fb9e442566c7ee9ee44739e4f88083bed37eaf3fe6eb34c361dc\": container with ID starting with fd47e37a8170fb9e442566c7ee9ee44739e4f88083bed37eaf3fe6eb34c361dc not found: ID does not exist" containerID="fd47e37a8170fb9e442566c7ee9ee44739e4f88083bed37eaf3fe6eb34c361dc" Sep 08 00:46:06.297798 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.297748 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd47e37a8170fb9e442566c7ee9ee44739e4f88083bed37eaf3fe6eb34c361dc"} err="failed to get container status \"fd47e37a8170fb9e442566c7ee9ee44739e4f88083bed37eaf3fe6eb34c361dc\": rpc error: code = NotFound desc = could not find container \"fd47e37a8170fb9e442566c7ee9ee44739e4f88083bed37eaf3fe6eb34c361dc\": container with ID starting with fd47e37a8170fb9e442566c7ee9ee44739e4f88083bed37eaf3fe6eb34c361dc not found: ID does not exist" Sep 08 00:46:06.305759 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.305737 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp"] Sep 08 00:46:06.310102 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.310079 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b001d57-75d47f6897-vlxmp"] Sep 08 00:46:06.319537 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.319518 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd"] Sep 08 00:46:06.326685 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.326665 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9b001d57-66454d55f8-wwwmd"] Sep 08 00:46:06.790939 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.790912 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="056ec851-30aa-44eb-b6b6-ace8a734d328" path="/var/lib/kubelet/pods/056ec851-30aa-44eb-b6b6-ace8a734d328/volumes" Sep 08 00:46:06.791385 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.791362 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec" path="/var/lib/kubelet/pods/e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec/volumes" Sep 08 00:46:06.791720 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:46:06.791704 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e86202e9-fe00-4974-8a68-70a551f24d84" path="/var/lib/kubelet/pods/e86202e9-fe00-4974-8a68-70a551f24d84/volumes" Sep 08 00:47:11.842452 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:11.842418 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j"] Sep 08 00:47:21.548922 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.548886 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9"] Sep 08 00:47:21.549446 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.549239 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0654cf5-8dd6-4180-b7f1-5b83cfc598b2" containerName="payload-pre-processing" Sep 08 00:47:21.549446 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.549252 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0654cf5-8dd6-4180-b7f1-5b83cfc598b2" containerName="payload-pre-processing" Sep 08 00:47:21.549446 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.549269 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec" containerName="revoke-keys" Sep 08 00:47:21.549446 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.549275 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec" containerName="revoke-keys" Sep 08 00:47:21.549446 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.549282 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056ec851-30aa-44eb-b6b6-ace8a734d328" containerName="payload-pre-processing" Sep 08 00:47:21.549446 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.549288 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="056ec851-30aa-44eb-b6b6-ace8a734d328" containerName="payload-pre-processing" Sep 08 00:47:21.549446 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.549298 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80f84a3d-f6fb-4067-bf08-629980db649b" containerName="payload-processing" Sep 08 00:47:21.549446 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.549304 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="80f84a3d-f6fb-4067-bf08-629980db649b" containerName="payload-processing" Sep 08 00:47:21.549446 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.549312 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e86202e9-fe00-4974-8a68-70a551f24d84" containerName="payload-processing" Sep 08 00:47:21.549446 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.549317 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e86202e9-fe00-4974-8a68-70a551f24d84" containerName="payload-processing" Sep 08 00:47:21.549446 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.549369 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0654cf5-8dd6-4180-b7f1-5b83cfc598b2" containerName="payload-pre-processing" Sep 08 00:47:21.549446 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.549375 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="056ec851-30aa-44eb-b6b6-ace8a734d328" containerName="payload-pre-processing" Sep 08 00:47:21.549446 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.549381 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2f96c56-8c6c-4a27-b6c3-35ade0ae16ec" containerName="revoke-keys" Sep 08 00:47:21.549446 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.549388 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="80f84a3d-f6fb-4067-bf08-629980db649b" containerName="payload-processing" Sep 08 00:47:21.549446 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.549395 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e86202e9-fe00-4974-8a68-70a551f24d84" containerName="payload-processing" Sep 08 00:47:21.552233 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.552217 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9" Sep 08 00:47:21.554698 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.554678 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-e4902f-dockercfg-xgcfj\"" Sep 08 00:47:21.554808 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.554716 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-e4902f\"" Sep 08 00:47:21.566789 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.566767 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9"] Sep 08 00:47:21.577396 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.577369 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35c6e2e-ade5-4723-99e8-d61a03fd3067-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9\" (UID: \"d35c6e2e-ade5-4723-99e8-d61a03fd3067\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9" Sep 08 00:47:21.577602 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.577577 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkrn9\" (UniqueName: \"kubernetes.io/projected/d35c6e2e-ade5-4723-99e8-d61a03fd3067-kube-api-access-bkrn9\") pod \"payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9\" (UID: \"d35c6e2e-ade5-4723-99e8-d61a03fd3067\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9" Sep 08 00:47:21.581310 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.581290 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh"] Sep 08 00:47:21.584707 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.584683 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh" Sep 08 00:47:21.601870 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.601842 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh"] Sep 08 00:47:21.678237 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.678198 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35c6e2e-ade5-4723-99e8-d61a03fd3067-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9\" (UID: \"d35c6e2e-ade5-4723-99e8-d61a03fd3067\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9" Sep 08 00:47:21.678420 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.678248 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dhwm\" (UniqueName: \"kubernetes.io/projected/ae761447-0122-4a4f-b176-1f8028aff6df-kube-api-access-5dhwm\") pod \"payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh\" (UID: \"ae761447-0122-4a4f-b176-1f8028aff6df\") " pod="openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh" Sep 08 00:47:21.678420 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.678277 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae761447-0122-4a4f-b176-1f8028aff6df-plugins-config-volume\") pod \"payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh\" (UID: \"ae761447-0122-4a4f-b176-1f8028aff6df\") " pod="openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh" Sep 08 00:47:21.678420 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.678296 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkrn9\" (UniqueName: \"kubernetes.io/projected/d35c6e2e-ade5-4723-99e8-d61a03fd3067-kube-api-access-bkrn9\") pod \"payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9\" (UID: \"d35c6e2e-ade5-4723-99e8-d61a03fd3067\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9" Sep 08 00:47:21.678977 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.678953 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35c6e2e-ade5-4723-99e8-d61a03fd3067-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9\" (UID: \"d35c6e2e-ade5-4723-99e8-d61a03fd3067\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9" Sep 08 00:47:21.686720 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.686689 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkrn9\" (UniqueName: \"kubernetes.io/projected/d35c6e2e-ade5-4723-99e8-d61a03fd3067-kube-api-access-bkrn9\") pod \"payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9\" (UID: \"d35c6e2e-ade5-4723-99e8-d61a03fd3067\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9" Sep 08 00:47:21.778655 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.778620 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dhwm\" (UniqueName: \"kubernetes.io/projected/ae761447-0122-4a4f-b176-1f8028aff6df-kube-api-access-5dhwm\") pod \"payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh\" (UID: \"ae761447-0122-4a4f-b176-1f8028aff6df\") " pod="openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh" Sep 08 00:47:21.778655 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.778661 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae761447-0122-4a4f-b176-1f8028aff6df-plugins-config-volume\") pod \"payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh\" (UID: \"ae761447-0122-4a4f-b176-1f8028aff6df\") " pod="openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh" Sep 08 00:47:21.779427 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.779406 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae761447-0122-4a4f-b176-1f8028aff6df-plugins-config-volume\") pod \"payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh\" (UID: \"ae761447-0122-4a4f-b176-1f8028aff6df\") " pod="openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh" Sep 08 00:47:21.788049 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.788021 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dhwm\" (UniqueName: \"kubernetes.io/projected/ae761447-0122-4a4f-b176-1f8028aff6df-kube-api-access-5dhwm\") pod \"payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh\" (UID: \"ae761447-0122-4a4f-b176-1f8028aff6df\") " pod="openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh" Sep 08 00:47:21.862119 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.862027 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9" Sep 08 00:47:21.893445 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:21.893413 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh" Sep 08 00:47:22.134487 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:22.134458 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9"] Sep 08 00:47:22.136251 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:47:22.136227 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35c6e2e_ade5_4723_99e8_d61a03fd3067.slice/crio-3854184c5aa57078bf1ce1e2908431bc2d3a06b38dfa5ea46680a4e9dbab89b0 WatchSource:0}: Error finding container 3854184c5aa57078bf1ce1e2908431bc2d3a06b38dfa5ea46680a4e9dbab89b0: Status 404 returned error can't find the container with id 3854184c5aa57078bf1ce1e2908431bc2d3a06b38dfa5ea46680a4e9dbab89b0 Sep 08 00:47:22.173075 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:22.173053 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh"] Sep 08 00:47:22.175900 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:47:22.175873 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae761447_0122_4a4f_b176_1f8028aff6df.slice/crio-3720414f83a60b821c48f7c568edc3e12f7d2345130fb92777ad58d584ece018 WatchSource:0}: Error finding container 3720414f83a60b821c48f7c568edc3e12f7d2345130fb92777ad58d584ece018: Status 404 returned error can't find the container with id 3720414f83a60b821c48f7c568edc3e12f7d2345130fb92777ad58d584ece018 Sep 08 00:47:22.647744 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:22.647692 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh" event={"ID":"ae761447-0122-4a4f-b176-1f8028aff6df","Type":"ContainerStarted","Data":"e667b2f3cd19f03460988f86539d8d3efbdd0a7d2542a81d5b62a640fa5a993c"} Sep 08 00:47:22.648226 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:22.647797 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh" event={"ID":"ae761447-0122-4a4f-b176-1f8028aff6df","Type":"ContainerStarted","Data":"3720414f83a60b821c48f7c568edc3e12f7d2345130fb92777ad58d584ece018"} Sep 08 00:47:22.648226 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:22.647828 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh" Sep 08 00:47:22.649287 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:22.649260 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9" event={"ID":"d35c6e2e-ade5-4723-99e8-d61a03fd3067","Type":"ContainerStarted","Data":"d1ed945d78b6160b84132fe18d29e723f5f55e5d75065bbcfa153f4575a9adf3"} Sep 08 00:47:22.649367 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:22.649308 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9" event={"ID":"d35c6e2e-ade5-4723-99e8-d61a03fd3067","Type":"ContainerStarted","Data":"3854184c5aa57078bf1ce1e2908431bc2d3a06b38dfa5ea46680a4e9dbab89b0"} Sep 08 00:47:22.649427 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:22.649363 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9" Sep 08 00:47:22.664592 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:22.664549 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh" podStartSLOduration=1.664536794 podStartE2EDuration="1.664536794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:47:22.662805153 +0000 UTC m=+856.358046314" watchObservedRunningTime="2026-09-08 00:47:22.664536794 +0000 UTC m=+856.359777972" Sep 08 00:47:22.678244 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:22.678197 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9" podStartSLOduration=1.6781846950000001 podStartE2EDuration="1.678184695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:47:22.678120762 +0000 UTC m=+856.373361922" watchObservedRunningTime="2026-09-08 00:47:22.678184695 +0000 UTC m=+856.373425855" Sep 08 00:47:26.862010 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:26.861958 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" podUID="6c3c0f13-0cc9-4ba7-9988-6422214e154f" containerName="main" containerID="cri-o://7a285185c7015ab59e9238758bbc0996541a195cb6fabfbd24a54a6ea7adbe47" gracePeriod=45 Sep 08 00:47:27.104321 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.104297 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:47:27.123821 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.123764 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-home\") pod \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " Sep 08 00:47:27.123821 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.123800 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-tmp-dir\") pod \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " Sep 08 00:47:27.124035 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.123831 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c3c0f13-0cc9-4ba7-9988-6422214e154f-tls-certs\") pod \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " Sep 08 00:47:27.124035 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.123879 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-dshm\") pod \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " Sep 08 00:47:27.124035 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.123924 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-model-cache\") pod \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " Sep 08 00:47:27.124035 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.123946 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtpcz\" (UniqueName: \"kubernetes.io/projected/6c3c0f13-0cc9-4ba7-9988-6422214e154f-kube-api-access-dtpcz\") pod \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\" (UID: \"6c3c0f13-0cc9-4ba7-9988-6422214e154f\") " Sep 08 00:47:27.124247 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.124081 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-home" (OuterVolumeSpecName: "home") pod "6c3c0f13-0cc9-4ba7-9988-6422214e154f" (UID: "6c3c0f13-0cc9-4ba7-9988-6422214e154f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:47:27.124247 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.124109 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c3c0f13-0cc9-4ba7-9988-6422214e154f" (UID: "6c3c0f13-0cc9-4ba7-9988-6422214e154f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:47:27.124247 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.124224 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c3c0f13-0cc9-4ba7-9988-6422214e154f" (UID: "6c3c0f13-0cc9-4ba7-9988-6422214e154f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:47:27.124385 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.124288 2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-home\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:47:27.124385 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.124306 2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-tmp-dir\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:47:27.126363 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.126297 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c3c0f13-0cc9-4ba7-9988-6422214e154f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c3c0f13-0cc9-4ba7-9988-6422214e154f" (UID: "6c3c0f13-0cc9-4ba7-9988-6422214e154f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 00:47:27.126363 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.126325 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-dshm" (OuterVolumeSpecName: "dshm") pod "6c3c0f13-0cc9-4ba7-9988-6422214e154f" (UID: "6c3c0f13-0cc9-4ba7-9988-6422214e154f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:47:27.126857 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.126834 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3c0f13-0cc9-4ba7-9988-6422214e154f-kube-api-access-dtpcz" (OuterVolumeSpecName: "kube-api-access-dtpcz") pod "6c3c0f13-0cc9-4ba7-9988-6422214e154f" (UID: "6c3c0f13-0cc9-4ba7-9988-6422214e154f"). InnerVolumeSpecName "kube-api-access-dtpcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:47:27.225549 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.225518 2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c3c0f13-0cc9-4ba7-9988-6422214e154f-tls-certs\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:47:27.225549 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.225546 2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-dshm\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:47:27.225549 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.225555 2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c3c0f13-0cc9-4ba7-9988-6422214e154f-model-cache\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:47:27.225774 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.225563 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtpcz\" (UniqueName: \"kubernetes.io/projected/6c3c0f13-0cc9-4ba7-9988-6422214e154f-kube-api-access-dtpcz\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:47:27.675115 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.675075 2577 generic.go:356] "Generic (PLEG): container finished" podID="6c3c0f13-0cc9-4ba7-9988-6422214e154f" containerID="7a285185c7015ab59e9238758bbc0996541a195cb6fabfbd24a54a6ea7adbe47" exitCode=0 Sep 08 00:47:27.675295 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.675188 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" Sep 08 00:47:27.675295 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.675195 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" event={"ID":"6c3c0f13-0cc9-4ba7-9988-6422214e154f","Type":"ContainerDied","Data":"7a285185c7015ab59e9238758bbc0996541a195cb6fabfbd24a54a6ea7adbe47"} Sep 08 00:47:27.675295 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.675230 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j" event={"ID":"6c3c0f13-0cc9-4ba7-9988-6422214e154f","Type":"ContainerDied","Data":"04570d728867fb955804adfaf1cbc2deb7524d4ed31211475a4becf3e7abfafa"} Sep 08 00:47:27.675295 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.675247 2577 scope.go:117] "RemoveContainer" containerID="7a285185c7015ab59e9238758bbc0996541a195cb6fabfbd24a54a6ea7adbe47" Sep 08 00:47:27.690119 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.690084 2577 scope.go:117] "RemoveContainer" containerID="7a285185c7015ab59e9238758bbc0996541a195cb6fabfbd24a54a6ea7adbe47" Sep 08 00:47:27.690478 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:47:27.690456 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a285185c7015ab59e9238758bbc0996541a195cb6fabfbd24a54a6ea7adbe47\": container with ID starting with 7a285185c7015ab59e9238758bbc0996541a195cb6fabfbd24a54a6ea7adbe47 not found: ID does not exist" containerID="7a285185c7015ab59e9238758bbc0996541a195cb6fabfbd24a54a6ea7adbe47" Sep 08 00:47:27.690546 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.690486 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a285185c7015ab59e9238758bbc0996541a195cb6fabfbd24a54a6ea7adbe47"} err="failed to get container status \"7a285185c7015ab59e9238758bbc0996541a195cb6fabfbd24a54a6ea7adbe47\": rpc error: code = NotFound desc = could not find container \"7a285185c7015ab59e9238758bbc0996541a195cb6fabfbd24a54a6ea7adbe47\": container with ID starting with 7a285185c7015ab59e9238758bbc0996541a195cb6fabfbd24a54a6ea7adbe47 not found: ID does not exist" Sep 08 00:47:27.699462 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.699435 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j"] Sep 08 00:47:27.703488 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:27.703463 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-4a8d93c0-kserve-77c864fbc8-frw9j"] Sep 08 00:47:28.790936 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:28.790906 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c3c0f13-0cc9-4ba7-9988-6422214e154f" path="/var/lib/kubelet/pods/6c3c0f13-0cc9-4ba7-9988-6422214e154f/volumes" Sep 08 00:47:33.656891 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:33.656862 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh" Sep 08 00:47:33.657287 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:33.656927 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9" Sep 08 00:47:45.695560 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:45.695524 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq"] Sep 08 00:47:45.695969 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:45.695843 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c3c0f13-0cc9-4ba7-9988-6422214e154f" containerName="main" Sep 08 00:47:45.695969 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:45.695854 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c3c0f13-0cc9-4ba7-9988-6422214e154f" containerName="main" Sep 08 00:47:45.695969 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:45.695917 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c3c0f13-0cc9-4ba7-9988-6422214e154f" containerName="main" Sep 08 00:47:45.698803 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:45.698783 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq" Sep 08 00:47:45.701500 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:45.701478 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f627f164-dockercfg-mnjhm\"" Sep 08 00:47:45.701588 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:45.701486 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f627f164\"" Sep 08 00:47:45.711111 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:45.711079 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq"] Sep 08 00:47:45.785645 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:45.785605 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwhqm\" (UniqueName: \"kubernetes.io/projected/f8d2d731-4156-4769-91ed-0b4c0edadaf7-kube-api-access-kwhqm\") pod \"payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq\" (UID: \"f8d2d731-4156-4769-91ed-0b4c0edadaf7\") " pod="openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq" Sep 08 00:47:45.785848 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:45.785674 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8d2d731-4156-4769-91ed-0b4c0edadaf7-plugins-config-volume\") pod \"payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq\" (UID: \"f8d2d731-4156-4769-91ed-0b4c0edadaf7\") " pod="openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq" Sep 08 00:47:45.886972 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:45.886934 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwhqm\" (UniqueName: \"kubernetes.io/projected/f8d2d731-4156-4769-91ed-0b4c0edadaf7-kube-api-access-kwhqm\") pod \"payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq\" (UID: \"f8d2d731-4156-4769-91ed-0b4c0edadaf7\") " pod="openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq" Sep 08 00:47:45.887193 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:45.886983 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8d2d731-4156-4769-91ed-0b4c0edadaf7-plugins-config-volume\") pod \"payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq\" (UID: \"f8d2d731-4156-4769-91ed-0b4c0edadaf7\") " pod="openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq" Sep 08 00:47:45.887668 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:45.887646 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8d2d731-4156-4769-91ed-0b4c0edadaf7-plugins-config-volume\") pod \"payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq\" (UID: \"f8d2d731-4156-4769-91ed-0b4c0edadaf7\") " pod="openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq" Sep 08 00:47:45.894924 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:45.894899 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwhqm\" (UniqueName: \"kubernetes.io/projected/f8d2d731-4156-4769-91ed-0b4c0edadaf7-kube-api-access-kwhqm\") pod \"payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq\" (UID: \"f8d2d731-4156-4769-91ed-0b4c0edadaf7\") " pod="openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq" Sep 08 00:47:46.009143 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:46.009048 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq" Sep 08 00:47:46.278929 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:46.278897 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq"] Sep 08 00:47:46.281396 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:47:46.281363 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d2d731_4156_4769_91ed_0b4c0edadaf7.slice/crio-7d43c27428cf3c618333291c40e082f62e41eb43aaec3b67d5d127e9e5c63627 WatchSource:0}: Error finding container 7d43c27428cf3c618333291c40e082f62e41eb43aaec3b67d5d127e9e5c63627: Status 404 returned error can't find the container with id 7d43c27428cf3c618333291c40e082f62e41eb43aaec3b67d5d127e9e5c63627 Sep 08 00:47:46.769918 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:46.769810 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq" event={"ID":"f8d2d731-4156-4769-91ed-0b4c0edadaf7","Type":"ContainerStarted","Data":"30ffc47437c009aa76ab4f0061ef716588fef28f8f580cbadafbf31864ed4f9f"} Sep 08 00:47:46.769918 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:46.769874 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq" event={"ID":"f8d2d731-4156-4769-91ed-0b4c0edadaf7","Type":"ContainerStarted","Data":"7d43c27428cf3c618333291c40e082f62e41eb43aaec3b67d5d127e9e5c63627"} Sep 08 00:47:46.769918 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:46.769902 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq" Sep 08 00:47:46.785556 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:46.785499 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq" podStartSLOduration=1.785480247 podStartE2EDuration="1.785480247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:47:46.785183066 +0000 UTC m=+880.480424226" watchObservedRunningTime="2026-09-08 00:47:46.785480247 +0000 UTC m=+880.480721408" Sep 08 00:47:57.777327 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:47:57.777290 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq" Sep 08 00:48:06.723454 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:06.723425 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9nb9x_a2e160ae-d75d-4959-a5ab-d238b4113736/console-operator/1.log" Sep 08 00:48:06.723892 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:06.723641 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9nb9x_a2e160ae-d75d-4959-a5ab-d238b4113736/console-operator/1.log" Sep 08 00:48:15.921935 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:15.921896 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f627f164-6rx2l"] Sep 08 00:48:15.926886 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:15.926867 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f627f164-6rx2l" Sep 08 00:48:15.930261 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:15.930029 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 00:48:15.930261 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:15.930046 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dqw2\"" Sep 08 00:48:15.931849 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:15.931823 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f627f164-6rx2l"] Sep 08 00:48:15.939913 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:15.939884 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c2da09-7c99-4d94-9c2c-1bc97a7094a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f627f164-6rx2l\" (UID: \"b3c2da09-7c99-4d94-9c2c-1bc97a7094a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f627f164-6rx2l" Sep 08 00:48:16.040787 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:16.040747 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c2da09-7c99-4d94-9c2c-1bc97a7094a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f627f164-6rx2l\" (UID: \"b3c2da09-7c99-4d94-9c2c-1bc97a7094a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f627f164-6rx2l" Sep 08 00:48:16.041367 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:16.041349 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c2da09-7c99-4d94-9c2c-1bc97a7094a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f627f164-6rx2l\" (UID: \"b3c2da09-7c99-4d94-9c2c-1bc97a7094a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f627f164-6rx2l" Sep 08 00:48:16.238737 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:16.238651 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f627f164-6rx2l" Sep 08 00:48:16.507279 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:16.507243 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f627f164-6rx2l"] Sep 08 00:48:16.509978 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:48:16.509951 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c2da09_7c99_4d94_9c2c_1bc97a7094a1.slice/crio-2f06efc2c0b89b4293176145e5fdbd16d7246bfffecf59cc66a6e3cb843b1c33 WatchSource:0}: Error finding container 2f06efc2c0b89b4293176145e5fdbd16d7246bfffecf59cc66a6e3cb843b1c33: Status 404 returned error can't find the container with id 2f06efc2c0b89b4293176145e5fdbd16d7246bfffecf59cc66a6e3cb843b1c33 Sep 08 00:48:16.931803 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:16.931762 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f627f164-6rx2l" event={"ID":"b3c2da09-7c99-4d94-9c2c-1bc97a7094a1","Type":"ContainerStarted","Data":"9d69747ff51d0a42cd4f05e1d55c0dd54079de7ca771b5cfa47b7f697fd3e11b"} Sep 08 00:48:16.932234 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:16.931816 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f627f164-6rx2l" event={"ID":"b3c2da09-7c99-4d94-9c2c-1bc97a7094a1","Type":"ContainerStarted","Data":"2f06efc2c0b89b4293176145e5fdbd16d7246bfffecf59cc66a6e3cb843b1c33"} Sep 08 00:48:16.950114 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:16.950062 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f627f164-6rx2l" podStartSLOduration=1.950047429 podStartE2EDuration="1.950047429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:48:16.948856758 +0000 UTC m=+910.644097923" watchObservedRunningTime="2026-09-08 00:48:16.950047429 +0000 UTC m=+910.645288587" Sep 08 00:48:17.943117 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:17.943076 2577 generic.go:356] "Generic (PLEG): container finished" podID="b3c2da09-7c99-4d94-9c2c-1bc97a7094a1" containerID="9d69747ff51d0a42cd4f05e1d55c0dd54079de7ca771b5cfa47b7f697fd3e11b" exitCode=0 Sep 08 00:48:17.943506 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:17.943159 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f627f164-6rx2l" event={"ID":"b3c2da09-7c99-4d94-9c2c-1bc97a7094a1","Type":"ContainerDied","Data":"9d69747ff51d0a42cd4f05e1d55c0dd54079de7ca771b5cfa47b7f697fd3e11b"} Sep 08 00:48:19.084573 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:19.084549 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f627f164-6rx2l" Sep 08 00:48:19.166850 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:19.166820 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c2da09-7c99-4d94-9c2c-1bc97a7094a1-maas-api-service-ca\") pod \"b3c2da09-7c99-4d94-9c2c-1bc97a7094a1\" (UID: \"b3c2da09-7c99-4d94-9c2c-1bc97a7094a1\") " Sep 08 00:48:19.167209 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:19.167185 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3c2da09-7c99-4d94-9c2c-1bc97a7094a1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b3c2da09-7c99-4d94-9c2c-1bc97a7094a1" (UID: "b3c2da09-7c99-4d94-9c2c-1bc97a7094a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:48:19.267627 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:19.267539 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3c2da09-7c99-4d94-9c2c-1bc97a7094a1-maas-api-service-ca\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:48:19.953706 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:19.953676 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f627f164-6rx2l" Sep 08 00:48:19.953885 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:19.953723 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f627f164-6rx2l" event={"ID":"b3c2da09-7c99-4d94-9c2c-1bc97a7094a1","Type":"ContainerDied","Data":"2f06efc2c0b89b4293176145e5fdbd16d7246bfffecf59cc66a6e3cb843b1c33"} Sep 08 00:48:19.953885 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:19.953771 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f06efc2c0b89b4293176145e5fdbd16d7246bfffecf59cc66a6e3cb843b1c33" Sep 08 00:48:26.536098 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:26.536066 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq"] Sep 08 00:48:26.536578 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:26.536282 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq" podUID="f8d2d731-4156-4769-91ed-0b4c0edadaf7" containerName="payload-processing" containerID="cri-o://30ffc47437c009aa76ab4f0061ef716588fef28f8f580cbadafbf31864ed4f9f" gracePeriod=30 Sep 08 00:48:26.708705 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:26.708678 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f627f164-6rx2l"] Sep 08 00:48:26.711382 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:26.711354 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f627f164-6rx2l"] Sep 08 00:48:26.781579 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:26.781552 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq" Sep 08 00:48:26.789942 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:26.789883 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3c2da09-7c99-4d94-9c2c-1bc97a7094a1" path="/var/lib/kubelet/pods/b3c2da09-7c99-4d94-9c2c-1bc97a7094a1/volumes" Sep 08 00:48:26.834252 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:26.834220 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwhqm\" (UniqueName: \"kubernetes.io/projected/f8d2d731-4156-4769-91ed-0b4c0edadaf7-kube-api-access-kwhqm\") pod \"f8d2d731-4156-4769-91ed-0b4c0edadaf7\" (UID: \"f8d2d731-4156-4769-91ed-0b4c0edadaf7\") " Sep 08 00:48:26.834386 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:26.834266 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8d2d731-4156-4769-91ed-0b4c0edadaf7-plugins-config-volume\") pod \"f8d2d731-4156-4769-91ed-0b4c0edadaf7\" (UID: \"f8d2d731-4156-4769-91ed-0b4c0edadaf7\") " Sep 08 00:48:26.834690 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:26.834634 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d2d731-4156-4769-91ed-0b4c0edadaf7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f8d2d731-4156-4769-91ed-0b4c0edadaf7" (UID: "f8d2d731-4156-4769-91ed-0b4c0edadaf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:48:26.836409 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:26.836379 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d2d731-4156-4769-91ed-0b4c0edadaf7-kube-api-access-kwhqm" (OuterVolumeSpecName: "kube-api-access-kwhqm") pod "f8d2d731-4156-4769-91ed-0b4c0edadaf7" (UID: "f8d2d731-4156-4769-91ed-0b4c0edadaf7"). InnerVolumeSpecName "kube-api-access-kwhqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:48:26.935304 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:26.935264 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8d2d731-4156-4769-91ed-0b4c0edadaf7-plugins-config-volume\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:48:26.935304 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:26.935299 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwhqm\" (UniqueName: \"kubernetes.io/projected/f8d2d731-4156-4769-91ed-0b4c0edadaf7-kube-api-access-kwhqm\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:48:26.988919 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:26.988885 2577 generic.go:356] "Generic (PLEG): container finished" podID="f8d2d731-4156-4769-91ed-0b4c0edadaf7" containerID="30ffc47437c009aa76ab4f0061ef716588fef28f8f580cbadafbf31864ed4f9f" exitCode=0 Sep 08 00:48:26.989103 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:26.988927 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq" Sep 08 00:48:26.989103 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:26.988960 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq" event={"ID":"f8d2d731-4156-4769-91ed-0b4c0edadaf7","Type":"ContainerDied","Data":"30ffc47437c009aa76ab4f0061ef716588fef28f8f580cbadafbf31864ed4f9f"} Sep 08 00:48:26.989103 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:26.989030 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq" event={"ID":"f8d2d731-4156-4769-91ed-0b4c0edadaf7","Type":"ContainerDied","Data":"7d43c27428cf3c618333291c40e082f62e41eb43aaec3b67d5d127e9e5c63627"} Sep 08 00:48:26.989103 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:26.989054 2577 scope.go:117] "RemoveContainer" containerID="30ffc47437c009aa76ab4f0061ef716588fef28f8f580cbadafbf31864ed4f9f" Sep 08 00:48:26.998096 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:26.998075 2577 scope.go:117] "RemoveContainer" containerID="30ffc47437c009aa76ab4f0061ef716588fef28f8f580cbadafbf31864ed4f9f" Sep 08 00:48:26.998343 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:48:26.998323 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30ffc47437c009aa76ab4f0061ef716588fef28f8f580cbadafbf31864ed4f9f\": container with ID starting with 30ffc47437c009aa76ab4f0061ef716588fef28f8f580cbadafbf31864ed4f9f not found: ID does not exist" containerID="30ffc47437c009aa76ab4f0061ef716588fef28f8f580cbadafbf31864ed4f9f" Sep 08 00:48:26.998383 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:26.998352 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30ffc47437c009aa76ab4f0061ef716588fef28f8f580cbadafbf31864ed4f9f"} err="failed to get container status \"30ffc47437c009aa76ab4f0061ef716588fef28f8f580cbadafbf31864ed4f9f\": rpc error: code = NotFound desc = could not find container \"30ffc47437c009aa76ab4f0061ef716588fef28f8f580cbadafbf31864ed4f9f\": container with ID starting with 30ffc47437c009aa76ab4f0061ef716588fef28f8f580cbadafbf31864ed4f9f not found: ID does not exist" Sep 08 00:48:27.009423 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:27.009400 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq"] Sep 08 00:48:27.014031 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:27.014011 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f627f164-6f9cc6f457-jqflq"] Sep 08 00:48:28.790150 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:28.790114 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8d2d731-4156-4769-91ed-0b4c0edadaf7" path="/var/lib/kubelet/pods/f8d2d731-4156-4769-91ed-0b4c0edadaf7/volumes" Sep 08 00:48:41.985087 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:41.985052 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e4902f-nfxd9"] Sep 08 00:48:41.985617 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:41.985565 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8d2d731-4156-4769-91ed-0b4c0edadaf7" containerName="payload-processing" Sep 08 00:48:41.985617 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:41.985587 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8d2d731-4156-4769-91ed-0b4c0edadaf7" containerName="payload-processing" Sep 08 00:48:41.985617 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:41.985613 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3c2da09-7c99-4d94-9c2c-1bc97a7094a1" containerName="revoke-keys" Sep 08 00:48:41.985746 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:41.985622 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3c2da09-7c99-4d94-9c2c-1bc97a7094a1" containerName="revoke-keys" Sep 08 00:48:41.985746 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:41.985687 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8d2d731-4156-4769-91ed-0b4c0edadaf7" containerName="payload-processing" Sep 08 00:48:41.985746 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:41.985694 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3c2da09-7c99-4d94-9c2c-1bc97a7094a1" containerName="revoke-keys" Sep 08 00:48:41.988573 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:41.988549 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e4902f-nfxd9" Sep 08 00:48:41.990981 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:41.990948 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 00:48:41.991113 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:41.991008 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dqw2\"" Sep 08 00:48:41.996483 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:41.996455 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e4902f-nfxd9"] Sep 08 00:48:42.060525 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:42.060483 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8437f06c-4a9e-48bb-942c-b24101038a74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-e4902f-nfxd9\" (UID: \"8437f06c-4a9e-48bb-942c-b24101038a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e4902f-nfxd9" Sep 08 00:48:42.161430 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:42.161386 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8437f06c-4a9e-48bb-942c-b24101038a74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-e4902f-nfxd9\" (UID: \"8437f06c-4a9e-48bb-942c-b24101038a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e4902f-nfxd9" Sep 08 00:48:42.162200 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:42.162165 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8437f06c-4a9e-48bb-942c-b24101038a74-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-e4902f-nfxd9\" (UID: \"8437f06c-4a9e-48bb-942c-b24101038a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e4902f-nfxd9" Sep 08 00:48:42.307568 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:42.307533 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e4902f-nfxd9" Sep 08 00:48:42.589359 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:42.589330 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e4902f-nfxd9"] Sep 08 00:48:42.591102 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:48:42.591073 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8437f06c_4a9e_48bb_942c_b24101038a74.slice/crio-d718888c0da121432b7d9eb9bfd4c316ea206be03c04a58c57ae028a19791a13 WatchSource:0}: Error finding container d718888c0da121432b7d9eb9bfd4c316ea206be03c04a58c57ae028a19791a13: Status 404 returned error can't find the container with id d718888c0da121432b7d9eb9bfd4c316ea206be03c04a58c57ae028a19791a13 Sep 08 00:48:43.067752 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:43.067712 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e4902f-nfxd9" event={"ID":"8437f06c-4a9e-48bb-942c-b24101038a74","Type":"ContainerStarted","Data":"ac3c06b3547ad4ba59f5be49340b91f0a0a2556dee632da681b253960b87a723"} Sep 08 00:48:43.068197 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:43.067788 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e4902f-nfxd9" event={"ID":"8437f06c-4a9e-48bb-942c-b24101038a74","Type":"ContainerStarted","Data":"d718888c0da121432b7d9eb9bfd4c316ea206be03c04a58c57ae028a19791a13"} Sep 08 00:48:43.083196 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:43.083149 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e4902f-nfxd9" podStartSLOduration=2.083134552 podStartE2EDuration="2.083134552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:48:43.081679836 +0000 UTC m=+936.776920995" watchObservedRunningTime="2026-09-08 00:48:43.083134552 +0000 UTC m=+936.778375742" Sep 08 00:48:44.075037 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:44.074977 2577 generic.go:356] "Generic (PLEG): container finished" podID="8437f06c-4a9e-48bb-942c-b24101038a74" containerID="ac3c06b3547ad4ba59f5be49340b91f0a0a2556dee632da681b253960b87a723" exitCode=0 Sep 08 00:48:44.075037 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:44.075021 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e4902f-nfxd9" event={"ID":"8437f06c-4a9e-48bb-942c-b24101038a74","Type":"ContainerDied","Data":"ac3c06b3547ad4ba59f5be49340b91f0a0a2556dee632da681b253960b87a723"} Sep 08 00:48:45.209759 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:45.209729 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e4902f-nfxd9" Sep 08 00:48:45.287936 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:45.287903 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8437f06c-4a9e-48bb-942c-b24101038a74-maas-api-service-ca\") pod \"8437f06c-4a9e-48bb-942c-b24101038a74\" (UID: \"8437f06c-4a9e-48bb-942c-b24101038a74\") " Sep 08 00:48:45.288238 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:45.288216 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8437f06c-4a9e-48bb-942c-b24101038a74-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8437f06c-4a9e-48bb-942c-b24101038a74" (UID: "8437f06c-4a9e-48bb-942c-b24101038a74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:48:45.388820 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:45.388740 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8437f06c-4a9e-48bb-942c-b24101038a74-maas-api-service-ca\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:48:46.086801 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:46.086776 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e4902f-nfxd9" event={"ID":"8437f06c-4a9e-48bb-942c-b24101038a74","Type":"ContainerDied","Data":"d718888c0da121432b7d9eb9bfd4c316ea206be03c04a58c57ae028a19791a13"} Sep 08 00:48:46.086906 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:46.086813 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e4902f-nfxd9" Sep 08 00:48:46.086906 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:46.086829 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d718888c0da121432b7d9eb9bfd4c316ea206be03c04a58c57ae028a19791a13" Sep 08 00:48:52.297169 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.297133 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt"] Sep 08 00:48:52.297532 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.297473 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8437f06c-4a9e-48bb-942c-b24101038a74" containerName="revoke-keys" Sep 08 00:48:52.297532 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.297485 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8437f06c-4a9e-48bb-942c-b24101038a74" containerName="revoke-keys" Sep 08 00:48:52.297638 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.297547 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8437f06c-4a9e-48bb-942c-b24101038a74" containerName="revoke-keys" Sep 08 00:48:52.300485 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.300466 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt" Sep 08 00:48:52.302887 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.302867 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-c99af4-dockercfg-7ngrl\"" Sep 08 00:48:52.303068 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.303050 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-c99af4\"" Sep 08 00:48:52.312470 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.312451 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt"] Sep 08 00:48:52.459105 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.459067 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8gf9\" (UniqueName: \"kubernetes.io/projected/85f65be3-8dcc-4373-a477-a619f34dc41a-kube-api-access-z8gf9\") pod \"payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt\" (UID: \"85f65be3-8dcc-4373-a477-a619f34dc41a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt" Sep 08 00:48:52.459251 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.459198 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f65be3-8dcc-4373-a477-a619f34dc41a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt\" (UID: \"85f65be3-8dcc-4373-a477-a619f34dc41a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt" Sep 08 00:48:52.501422 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.501393 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh"] Sep 08 00:48:52.501665 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.501636 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh" podUID="ae761447-0122-4a4f-b176-1f8028aff6df" containerName="payload-processing" containerID="cri-o://e667b2f3cd19f03460988f86539d8d3efbdd0a7d2542a81d5b62a640fa5a993c" gracePeriod=30 Sep 08 00:48:52.508744 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.508709 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9"] Sep 08 00:48:52.509109 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.508914 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9" podUID="d35c6e2e-ade5-4723-99e8-d61a03fd3067" containerName="payload-pre-processing" containerID="cri-o://d1ed945d78b6160b84132fe18d29e723f5f55e5d75065bbcfa153f4575a9adf3" gracePeriod=30 Sep 08 00:48:52.559674 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.559593 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f65be3-8dcc-4373-a477-a619f34dc41a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt\" (UID: \"85f65be3-8dcc-4373-a477-a619f34dc41a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt" Sep 08 00:48:52.559674 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.559666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8gf9\" (UniqueName: \"kubernetes.io/projected/85f65be3-8dcc-4373-a477-a619f34dc41a-kube-api-access-z8gf9\") pod \"payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt\" (UID: \"85f65be3-8dcc-4373-a477-a619f34dc41a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt" Sep 08 00:48:52.560320 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.560296 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f65be3-8dcc-4373-a477-a619f34dc41a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt\" (UID: \"85f65be3-8dcc-4373-a477-a619f34dc41a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt" Sep 08 00:48:52.568265 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.568240 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8gf9\" (UniqueName: \"kubernetes.io/projected/85f65be3-8dcc-4373-a477-a619f34dc41a-kube-api-access-z8gf9\") pod \"payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt\" (UID: \"85f65be3-8dcc-4373-a477-a619f34dc41a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt" Sep 08 00:48:52.611061 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.611038 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt" Sep 08 00:48:52.699296 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.699265 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e4902f-nfxd9"] Sep 08 00:48:52.701848 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.701824 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-e4902f-nfxd9"] Sep 08 00:48:52.792037 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.792009 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8437f06c-4a9e-48bb-942c-b24101038a74" path="/var/lib/kubelet/pods/8437f06c-4a9e-48bb-942c-b24101038a74/volumes" Sep 08 00:48:52.804140 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.804119 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh" Sep 08 00:48:52.806060 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.806045 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9" Sep 08 00:48:52.884303 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.884279 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt"] Sep 08 00:48:52.886676 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:48:52.886648 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f65be3_8dcc_4373_a477_a619f34dc41a.slice/crio-53d6caa4678543cd750235c105875c12af03ad4e9f30dc68d6c4cad483ab4c61 WatchSource:0}: Error finding container 53d6caa4678543cd750235c105875c12af03ad4e9f30dc68d6c4cad483ab4c61: Status 404 returned error can't find the container with id 53d6caa4678543cd750235c105875c12af03ad4e9f30dc68d6c4cad483ab4c61 Sep 08 00:48:52.962676 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.962652 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkrn9\" (UniqueName: \"kubernetes.io/projected/d35c6e2e-ade5-4723-99e8-d61a03fd3067-kube-api-access-bkrn9\") pod \"d35c6e2e-ade5-4723-99e8-d61a03fd3067\" (UID: \"d35c6e2e-ade5-4723-99e8-d61a03fd3067\") " Sep 08 00:48:52.962790 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.962714 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dhwm\" (UniqueName: \"kubernetes.io/projected/ae761447-0122-4a4f-b176-1f8028aff6df-kube-api-access-5dhwm\") pod \"ae761447-0122-4a4f-b176-1f8028aff6df\" (UID: \"ae761447-0122-4a4f-b176-1f8028aff6df\") " Sep 08 00:48:52.962790 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.962737 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae761447-0122-4a4f-b176-1f8028aff6df-plugins-config-volume\") pod \"ae761447-0122-4a4f-b176-1f8028aff6df\" (UID: \"ae761447-0122-4a4f-b176-1f8028aff6df\") " Sep 08 00:48:52.962790 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.962760 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35c6e2e-ade5-4723-99e8-d61a03fd3067-plugins-config-volume\") pod \"d35c6e2e-ade5-4723-99e8-d61a03fd3067\" (UID: \"d35c6e2e-ade5-4723-99e8-d61a03fd3067\") " Sep 08 00:48:52.963193 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.963163 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae761447-0122-4a4f-b176-1f8028aff6df-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ae761447-0122-4a4f-b176-1f8028aff6df" (UID: "ae761447-0122-4a4f-b176-1f8028aff6df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:48:52.963328 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.963167 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35c6e2e-ade5-4723-99e8-d61a03fd3067-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d35c6e2e-ade5-4723-99e8-d61a03fd3067" (UID: "d35c6e2e-ade5-4723-99e8-d61a03fd3067"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:48:52.964642 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.964617 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35c6e2e-ade5-4723-99e8-d61a03fd3067-kube-api-access-bkrn9" (OuterVolumeSpecName: "kube-api-access-bkrn9") pod "d35c6e2e-ade5-4723-99e8-d61a03fd3067" (UID: "d35c6e2e-ade5-4723-99e8-d61a03fd3067"). InnerVolumeSpecName "kube-api-access-bkrn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:48:52.964778 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:52.964761 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae761447-0122-4a4f-b176-1f8028aff6df-kube-api-access-5dhwm" (OuterVolumeSpecName: "kube-api-access-5dhwm") pod "ae761447-0122-4a4f-b176-1f8028aff6df" (UID: "ae761447-0122-4a4f-b176-1f8028aff6df"). InnerVolumeSpecName "kube-api-access-5dhwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:48:53.064300 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.064263 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkrn9\" (UniqueName: \"kubernetes.io/projected/d35c6e2e-ade5-4723-99e8-d61a03fd3067-kube-api-access-bkrn9\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:48:53.064300 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.064294 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dhwm\" (UniqueName: \"kubernetes.io/projected/ae761447-0122-4a4f-b176-1f8028aff6df-kube-api-access-5dhwm\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:48:53.064300 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.064307 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae761447-0122-4a4f-b176-1f8028aff6df-plugins-config-volume\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:48:53.064768 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.064321 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d35c6e2e-ade5-4723-99e8-d61a03fd3067-plugins-config-volume\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:48:53.124517 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.124463 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt" event={"ID":"85f65be3-8dcc-4373-a477-a619f34dc41a","Type":"ContainerStarted","Data":"783c470181aebaa1098ca61ec9c7bc7a7f7c15d90c3f8dc43a5a3c100afe1ac1"} Sep 08 00:48:53.124707 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.124549 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt" Sep 08 00:48:53.124707 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.124604 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt" event={"ID":"85f65be3-8dcc-4373-a477-a619f34dc41a","Type":"ContainerStarted","Data":"53d6caa4678543cd750235c105875c12af03ad4e9f30dc68d6c4cad483ab4c61"} Sep 08 00:48:53.125760 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.125730 2577 generic.go:356] "Generic (PLEG): container finished" podID="ae761447-0122-4a4f-b176-1f8028aff6df" containerID="e667b2f3cd19f03460988f86539d8d3efbdd0a7d2542a81d5b62a640fa5a993c" exitCode=0 Sep 08 00:48:53.125877 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.125759 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh" Sep 08 00:48:53.125877 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.125808 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh" event={"ID":"ae761447-0122-4a4f-b176-1f8028aff6df","Type":"ContainerDied","Data":"e667b2f3cd19f03460988f86539d8d3efbdd0a7d2542a81d5b62a640fa5a993c"} Sep 08 00:48:53.125877 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.125846 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh" event={"ID":"ae761447-0122-4a4f-b176-1f8028aff6df","Type":"ContainerDied","Data":"3720414f83a60b821c48f7c568edc3e12f7d2345130fb92777ad58d584ece018"} Sep 08 00:48:53.126057 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.125883 2577 scope.go:117] "RemoveContainer" containerID="e667b2f3cd19f03460988f86539d8d3efbdd0a7d2542a81d5b62a640fa5a993c" Sep 08 00:48:53.127164 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.127138 2577 generic.go:356] "Generic (PLEG): container finished" podID="d35c6e2e-ade5-4723-99e8-d61a03fd3067" containerID="d1ed945d78b6160b84132fe18d29e723f5f55e5d75065bbcfa153f4575a9adf3" exitCode=0 Sep 08 00:48:53.127258 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.127174 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9" event={"ID":"d35c6e2e-ade5-4723-99e8-d61a03fd3067","Type":"ContainerDied","Data":"d1ed945d78b6160b84132fe18d29e723f5f55e5d75065bbcfa153f4575a9adf3"} Sep 08 00:48:53.127258 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.127213 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9" event={"ID":"d35c6e2e-ade5-4723-99e8-d61a03fd3067","Type":"ContainerDied","Data":"3854184c5aa57078bf1ce1e2908431bc2d3a06b38dfa5ea46680a4e9dbab89b0"} Sep 08 00:48:53.127258 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.127175 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9" Sep 08 00:48:53.134506 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.134488 2577 scope.go:117] "RemoveContainer" containerID="e667b2f3cd19f03460988f86539d8d3efbdd0a7d2542a81d5b62a640fa5a993c" Sep 08 00:48:53.134805 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:48:53.134775 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e667b2f3cd19f03460988f86539d8d3efbdd0a7d2542a81d5b62a640fa5a993c\": container with ID starting with e667b2f3cd19f03460988f86539d8d3efbdd0a7d2542a81d5b62a640fa5a993c not found: ID does not exist" containerID="e667b2f3cd19f03460988f86539d8d3efbdd0a7d2542a81d5b62a640fa5a993c" Sep 08 00:48:53.134880 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.134829 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e667b2f3cd19f03460988f86539d8d3efbdd0a7d2542a81d5b62a640fa5a993c"} err="failed to get container status \"e667b2f3cd19f03460988f86539d8d3efbdd0a7d2542a81d5b62a640fa5a993c\": rpc error: code = NotFound desc = could not find container \"e667b2f3cd19f03460988f86539d8d3efbdd0a7d2542a81d5b62a640fa5a993c\": container with ID starting with e667b2f3cd19f03460988f86539d8d3efbdd0a7d2542a81d5b62a640fa5a993c not found: ID does not exist" Sep 08 00:48:53.134880 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.134849 2577 scope.go:117] "RemoveContainer" containerID="d1ed945d78b6160b84132fe18d29e723f5f55e5d75065bbcfa153f4575a9adf3" Sep 08 00:48:53.143717 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.143700 2577 scope.go:117] "RemoveContainer" containerID="d1ed945d78b6160b84132fe18d29e723f5f55e5d75065bbcfa153f4575a9adf3" Sep 08 00:48:53.143967 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:48:53.143949 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ed945d78b6160b84132fe18d29e723f5f55e5d75065bbcfa153f4575a9adf3\": container with ID starting with d1ed945d78b6160b84132fe18d29e723f5f55e5d75065bbcfa153f4575a9adf3 not found: ID does not exist" containerID="d1ed945d78b6160b84132fe18d29e723f5f55e5d75065bbcfa153f4575a9adf3" Sep 08 00:48:53.144035 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.143973 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ed945d78b6160b84132fe18d29e723f5f55e5d75065bbcfa153f4575a9adf3"} err="failed to get container status \"d1ed945d78b6160b84132fe18d29e723f5f55e5d75065bbcfa153f4575a9adf3\": rpc error: code = NotFound desc = could not find container \"d1ed945d78b6160b84132fe18d29e723f5f55e5d75065bbcfa153f4575a9adf3\": container with ID starting with d1ed945d78b6160b84132fe18d29e723f5f55e5d75065bbcfa153f4575a9adf3 not found: ID does not exist" Sep 08 00:48:53.173687 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.173624 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt" podStartSLOduration=1.173602381 podStartE2EDuration="1.173602381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:48:53.172755643 +0000 UTC m=+946.867996801" watchObservedRunningTime="2026-09-08 00:48:53.173602381 +0000 UTC m=+946.868843541" Sep 08 00:48:53.200390 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.200355 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9"] Sep 08 00:48:53.212118 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.212083 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-e4902f-5c5fbfd878-f5sn9"] Sep 08 00:48:53.244724 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.240701 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh"] Sep 08 00:48:53.259617 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:53.259584 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-e4902f-77b76dd5b9-4xfdh"] Sep 08 00:48:54.790255 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:54.790219 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae761447-0122-4a4f-b176-1f8028aff6df" path="/var/lib/kubelet/pods/ae761447-0122-4a4f-b176-1f8028aff6df/volumes" Sep 08 00:48:54.790634 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:54.790556 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d35c6e2e-ade5-4723-99e8-d61a03fd3067" path="/var/lib/kubelet/pods/d35c6e2e-ade5-4723-99e8-d61a03fd3067/volumes" Sep 08 00:48:59.928580 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:59.928542 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n"] Sep 08 00:48:59.929061 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:59.929037 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d35c6e2e-ade5-4723-99e8-d61a03fd3067" containerName="payload-pre-processing" Sep 08 00:48:59.929061 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:59.929055 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d35c6e2e-ade5-4723-99e8-d61a03fd3067" containerName="payload-pre-processing" Sep 08 00:48:59.929198 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:59.929079 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae761447-0122-4a4f-b176-1f8028aff6df" containerName="payload-processing" Sep 08 00:48:59.929198 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:59.929089 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae761447-0122-4a4f-b176-1f8028aff6df" containerName="payload-processing" Sep 08 00:48:59.929198 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:59.929190 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae761447-0122-4a4f-b176-1f8028aff6df" containerName="payload-processing" Sep 08 00:48:59.929347 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:59.929203 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d35c6e2e-ade5-4723-99e8-d61a03fd3067" containerName="payload-pre-processing" Sep 08 00:48:59.933543 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:59.933522 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:48:59.935905 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:59.935884 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-75c97b-openshift-default-dockercfg-4n7x7\"" Sep 08 00:48:59.945055 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:48:59.945032 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n"] Sep 08 00:49:00.030112 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.030075 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-data\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.030358 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.030334 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd62e91-ec87-42fd-93f1-64d112af1ede-istiod-ca-cert\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.030564 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.030548 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-workload-socket\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.030685 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.030672 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-credential-socket\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.030808 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.030795 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-workload-certs\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.030937 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.030923 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-podinfo\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.031094 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.031078 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-token\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.031210 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.031195 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdr5g\" (UniqueName: \"kubernetes.io/projected/bbd62e91-ec87-42fd-93f1-64d112af1ede-kube-api-access-wdr5g\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.031360 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.031346 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-envoy\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.132243 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.132196 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-workload-socket\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.132243 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.132250 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-credential-socket\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.132499 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.132277 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-workload-certs\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.132499 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.132313 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-podinfo\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.132499 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.132349 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-token\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.132499 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.132370 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdr5g\" (UniqueName: \"kubernetes.io/projected/bbd62e91-ec87-42fd-93f1-64d112af1ede-kube-api-access-wdr5g\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.132499 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.132418 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-envoy\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.132499 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.132456 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-data\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.132499 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.132483 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd62e91-ec87-42fd-93f1-64d112af1ede-istiod-ca-cert\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.132838 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.132663 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-workload-socket\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.132838 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.132721 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-credential-socket\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.133026 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.132973 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-workload-certs\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.133026 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.132991 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-data\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.133246 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.133227 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd62e91-ec87-42fd-93f1-64d112af1ede-istiod-ca-cert\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.134667 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.134645 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-envoy\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.134780 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.134763 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-podinfo\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.139897 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.139871 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdr5g\" (UniqueName: \"kubernetes.io/projected/bbd62e91-ec87-42fd-93f1-64d112af1ede-kube-api-access-wdr5g\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.139897 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.139893 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-token\") pod \"e2e-noten-75c97b-openshift-default-66689f74b-gnj5n\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.245049 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.244946 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:00.520629 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.520604 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n"] Sep 08 00:49:00.523181 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:49:00.523153 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd62e91_ec87_42fd_93f1_64d112af1ede.slice/crio-23524525f739ade6761bb1c47cdccbaa14761620f80af7ac82526181ff35dabe WatchSource:0}: Error finding container 23524525f739ade6761bb1c47cdccbaa14761620f80af7ac82526181ff35dabe: Status 404 returned error can't find the container with id 23524525f739ade6761bb1c47cdccbaa14761620f80af7ac82526181ff35dabe Sep 08 00:49:00.525385 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.525323 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:49:00.525487 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.525430 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:49:00.525487 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:00.525473 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:49:01.171582 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:01.171548 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" event={"ID":"bbd62e91-ec87-42fd-93f1-64d112af1ede","Type":"ContainerStarted","Data":"0fae84923e17a1479c91aa1b21cba9580fadb2c67634b191b3b88eabad799639"} Sep 08 00:49:01.171966 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:01.171599 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" event={"ID":"bbd62e91-ec87-42fd-93f1-64d112af1ede","Type":"ContainerStarted","Data":"23524525f739ade6761bb1c47cdccbaa14761620f80af7ac82526181ff35dabe"} Sep 08 00:49:01.191679 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:01.191630 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" podStartSLOduration=2.191615586 podStartE2EDuration="2.191615586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:49:01.190146115 +0000 UTC m=+954.885387275" watchObservedRunningTime="2026-09-08 00:49:01.191615586 +0000 UTC m=+954.886856814" Sep 08 00:49:01.245660 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:01.245634 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:01.250469 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:01.250443 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:02.176337 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:02.176303 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:02.177292 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:02.177273 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:04.137425 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:04.137387 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt" Sep 08 00:49:15.756306 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.756273 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq"] Sep 08 00:49:15.761031 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.760986 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.765588 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.765568 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-f04ef7-openshift-default-dockercfg-8dx4r\"" Sep 08 00:49:15.774662 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.774633 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq"] Sep 08 00:49:15.869905 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.869871 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-workload-socket\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.870125 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.869910 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-token\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.870125 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.869944 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-data\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.870125 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.869963 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb6451f-d144-45b9-a925-bf63f81ff56f-istiod-ca-cert\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.870125 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.870023 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-www2v\" (UniqueName: \"kubernetes.io/projected/1fb6451f-d144-45b9-a925-bf63f81ff56f-kube-api-access-www2v\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.870125 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.870067 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-podinfo\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.870125 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.870096 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-envoy\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.870370 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.870141 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-workload-certs\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.870370 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.870172 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-credential-socket\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.971202 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.971166 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-www2v\" (UniqueName: \"kubernetes.io/projected/1fb6451f-d144-45b9-a925-bf63f81ff56f-kube-api-access-www2v\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.971414 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.971220 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-podinfo\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.971414 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.971250 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-envoy\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.971414 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.971299 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-workload-certs\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.971414 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.971329 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-credential-socket\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.971414 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.971408 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-workload-socket\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.971698 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.971432 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-token\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.971698 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.971467 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-data\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.971698 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.971491 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb6451f-d144-45b9-a925-bf63f81ff56f-istiod-ca-cert\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.972082 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.971768 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-workload-certs\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.972082 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.971816 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-credential-socket\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.972082 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.971906 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-workload-socket\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.972256 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.972097 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-data\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.972319 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.972262 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb6451f-d144-45b9-a925-bf63f81ff56f-istiod-ca-cert\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.974183 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.974158 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-envoy\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.974381 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.974361 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-podinfo\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.979246 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.979219 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-token\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:15.979336 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:15.979259 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-www2v\" (UniqueName: \"kubernetes.io/projected/1fb6451f-d144-45b9-a925-bf63f81ff56f-kube-api-access-www2v\") pod \"e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:16.072062 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:16.072028 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:16.344422 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:16.344177 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq"] Sep 08 00:49:16.346919 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:49:16.346888 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb6451f_d144_45b9_a925_bf63f81ff56f.slice/crio-21287d37f455411ab38609299f42816af3de55e707f1506938d9952a8a117c38 WatchSource:0}: Error finding container 21287d37f455411ab38609299f42816af3de55e707f1506938d9952a8a117c38: Status 404 returned error can't find the container with id 21287d37f455411ab38609299f42816af3de55e707f1506938d9952a8a117c38 Sep 08 00:49:16.349136 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:16.349101 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:49:16.349220 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:16.349169 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:49:16.349220 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:16.349195 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:49:17.249363 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:17.249324 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" event={"ID":"1fb6451f-d144-45b9-a925-bf63f81ff56f","Type":"ContainerStarted","Data":"2aeba9c454e9e80b940e79321d5233671792733bf354ce26a533c81f37cd0384"} Sep 08 00:49:17.249755 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:17.249385 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" event={"ID":"1fb6451f-d144-45b9-a925-bf63f81ff56f","Type":"ContainerStarted","Data":"21287d37f455411ab38609299f42816af3de55e707f1506938d9952a8a117c38"} Sep 08 00:49:17.269646 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:17.269597 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" podStartSLOduration=2.269581523 podStartE2EDuration="2.269581523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:49:17.268589632 +0000 UTC m=+970.963830818" watchObservedRunningTime="2026-09-08 00:49:17.269581523 +0000 UTC m=+970.964822681" Sep 08 00:49:18.072161 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:18.072124 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:18.077280 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:18.077253 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:18.254640 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:18.254613 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:18.255669 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:18.255645 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:38.355474 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:38.355439 2577 generic.go:356] "Generic (PLEG): container finished" podID="1fb6451f-d144-45b9-a925-bf63f81ff56f" containerID="2aeba9c454e9e80b940e79321d5233671792733bf354ce26a533c81f37cd0384" exitCode=0 Sep 08 00:49:38.355888 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:38.355512 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" event={"ID":"1fb6451f-d144-45b9-a925-bf63f81ff56f","Type":"ContainerDied","Data":"2aeba9c454e9e80b940e79321d5233671792733bf354ce26a533c81f37cd0384"} Sep 08 00:49:38.355888 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:38.355877 2577 scope.go:117] "RemoveContainer" containerID="2aeba9c454e9e80b940e79321d5233671792733bf354ce26a533c81f37cd0384" Sep 08 00:49:38.357024 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:38.356972 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:49:38.357081 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:38.357066 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:49:38.357121 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:38.357097 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:49:39.361954 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:39.361919 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" event={"ID":"1fb6451f-d144-45b9-a925-bf63f81ff56f","Type":"ContainerStarted","Data":"9ba5a7bd685c5512fc81b16622116df85753a41f5425ebf1a9e35802f1df0305"} Sep 08 00:49:40.073008 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:40.072951 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:40.073260 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:40.073236 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 08 00:49:40.073322 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:40.073302 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" podUID="1fb6451f-d144-45b9-a925-bf63f81ff56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 08 00:49:40.953394 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:40.953361 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n"] Sep 08 00:49:40.953854 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:40.953606 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" podUID="bbd62e91-ec87-42fd-93f1-64d112af1ede" containerName="istio-proxy" containerID="cri-o://0fae84923e17a1479c91aa1b21cba9580fadb2c67634b191b3b88eabad799639" gracePeriod=30 Sep 08 00:49:41.073120 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:41.073079 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 08 00:49:41.073317 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:41.073142 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" podUID="1fb6451f-d144-45b9-a925-bf63f81ff56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 08 00:49:42.073152 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:42.073109 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 08 00:49:42.073623 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:42.073175 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" podUID="1fb6451f-d144-45b9-a925-bf63f81ff56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 08 00:49:43.072692 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:43.072652 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 08 00:49:43.072869 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:43.072723 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" podUID="1fb6451f-d144-45b9-a925-bf63f81ff56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 08 00:49:43.635804 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:43.635764 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1cf84f-x8bsq"] Sep 08 00:49:43.639306 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:43.639277 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1cf84f-x8bsq" Sep 08 00:49:43.642098 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:43.642070 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 00:49:43.642309 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:43.642072 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dqw2\"" Sep 08 00:49:43.646274 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:43.646254 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1cf84f-x8bsq"] Sep 08 00:49:43.739307 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:43.739267 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/335cee51-973f-4378-b212-28e555e71523-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1cf84f-x8bsq\" (UID: \"335cee51-973f-4378-b212-28e555e71523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1cf84f-x8bsq" Sep 08 00:49:43.840723 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:43.840690 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/335cee51-973f-4378-b212-28e555e71523-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1cf84f-x8bsq\" (UID: \"335cee51-973f-4378-b212-28e555e71523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1cf84f-x8bsq" Sep 08 00:49:43.841350 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:43.841330 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/335cee51-973f-4378-b212-28e555e71523-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1cf84f-x8bsq\" (UID: \"335cee51-973f-4378-b212-28e555e71523\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1cf84f-x8bsq" Sep 08 00:49:43.953022 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:43.952904 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1cf84f-x8bsq" Sep 08 00:49:44.072489 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:44.072452 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 08 00:49:44.072695 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:44.072519 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" podUID="1fb6451f-d144-45b9-a925-bf63f81ff56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 08 00:49:44.246808 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:44.246785 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1cf84f-x8bsq"] Sep 08 00:49:44.248859 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:49:44.248830 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335cee51_973f_4378_b212_28e555e71523.slice/crio-e3950ceee8ab9c36bae88b0148b15dda905cab4e9e6ea5ecc8327604591204e1 WatchSource:0}: Error finding container e3950ceee8ab9c36bae88b0148b15dda905cab4e9e6ea5ecc8327604591204e1: Status 404 returned error can't find the container with id e3950ceee8ab9c36bae88b0148b15dda905cab4e9e6ea5ecc8327604591204e1 Sep 08 00:49:44.410020 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:44.406249 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1cf84f-x8bsq" event={"ID":"335cee51-973f-4378-b212-28e555e71523","Type":"ContainerStarted","Data":"e3950ceee8ab9c36bae88b0148b15dda905cab4e9e6ea5ecc8327604591204e1"} Sep 08 00:49:45.073212 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:45.073175 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 08 00:49:45.073608 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:45.073227 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" podUID="1fb6451f-d144-45b9-a925-bf63f81ff56f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 08 00:49:45.413186 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:45.413096 2577 generic.go:356] "Generic (PLEG): container finished" podID="335cee51-973f-4378-b212-28e555e71523" containerID="55503a902fa3d23b46abcee23454ebd643c89811346406e69de0829ad2ba8c43" exitCode=0 Sep 08 00:49:45.413186 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:45.413147 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1cf84f-x8bsq" event={"ID":"335cee51-973f-4378-b212-28e555e71523","Type":"ContainerDied","Data":"55503a902fa3d23b46abcee23454ebd643c89811346406e69de0829ad2ba8c43"} Sep 08 00:49:46.078036 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.077991 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:46.078462 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.078439 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:46.079278 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.079257 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:49:46.205798 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.205773 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:46.262531 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.262458 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd62e91-ec87-42fd-93f1-64d112af1ede-istiod-ca-cert\") pod \"bbd62e91-ec87-42fd-93f1-64d112af1ede\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " Sep 08 00:49:46.262531 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.262505 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-data\") pod \"bbd62e91-ec87-42fd-93f1-64d112af1ede\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " Sep 08 00:49:46.262755 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.262545 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdr5g\" (UniqueName: \"kubernetes.io/projected/bbd62e91-ec87-42fd-93f1-64d112af1ede-kube-api-access-wdr5g\") pod \"bbd62e91-ec87-42fd-93f1-64d112af1ede\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " Sep 08 00:49:46.262755 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.262567 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-podinfo\") pod \"bbd62e91-ec87-42fd-93f1-64d112af1ede\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " Sep 08 00:49:46.262755 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.262630 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-envoy\") pod \"bbd62e91-ec87-42fd-93f1-64d112af1ede\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " Sep 08 00:49:46.262755 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.262654 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-workload-certs\") pod \"bbd62e91-ec87-42fd-93f1-64d112af1ede\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " Sep 08 00:49:46.262974 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.262774 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-credential-socket\") pod \"bbd62e91-ec87-42fd-93f1-64d112af1ede\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " Sep 08 00:49:46.262974 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.262842 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-workload-socket\") pod \"bbd62e91-ec87-42fd-93f1-64d112af1ede\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " Sep 08 00:49:46.262974 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.262877 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-token\") pod \"bbd62e91-ec87-42fd-93f1-64d112af1ede\" (UID: \"bbd62e91-ec87-42fd-93f1-64d112af1ede\") " Sep 08 00:49:46.262974 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.262915 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-data" (OuterVolumeSpecName: "istio-data") pod "bbd62e91-ec87-42fd-93f1-64d112af1ede" (UID: "bbd62e91-ec87-42fd-93f1-64d112af1ede"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:49:46.262974 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.262927 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bbd62e91-ec87-42fd-93f1-64d112af1ede" (UID: "bbd62e91-ec87-42fd-93f1-64d112af1ede"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:49:46.262974 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.262931 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd62e91-ec87-42fd-93f1-64d112af1ede-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bbd62e91-ec87-42fd-93f1-64d112af1ede" (UID: "bbd62e91-ec87-42fd-93f1-64d112af1ede"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:49:46.263306 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.263113 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bbd62e91-ec87-42fd-93f1-64d112af1ede" (UID: "bbd62e91-ec87-42fd-93f1-64d112af1ede"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:49:46.263306 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.263133 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bbd62e91-ec87-42fd-93f1-64d112af1ede" (UID: "bbd62e91-ec87-42fd-93f1-64d112af1ede"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:49:46.263306 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.263238 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-workload-certs\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:49:46.263306 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.263257 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-credential-socket\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:49:46.263306 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.263268 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-workload-socket\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:49:46.263306 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.263277 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd62e91-ec87-42fd-93f1-64d112af1ede-istiod-ca-cert\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:49:46.263306 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.263285 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-data\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:49:46.264945 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.264921 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bbd62e91-ec87-42fd-93f1-64d112af1ede" (UID: "bbd62e91-ec87-42fd-93f1-64d112af1ede"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 00:49:46.265087 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.264925 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bbd62e91-ec87-42fd-93f1-64d112af1ede" (UID: "bbd62e91-ec87-42fd-93f1-64d112af1ede"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:49:46.265276 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.265252 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd62e91-ec87-42fd-93f1-64d112af1ede-kube-api-access-wdr5g" (OuterVolumeSpecName: "kube-api-access-wdr5g") pod "bbd62e91-ec87-42fd-93f1-64d112af1ede" (UID: "bbd62e91-ec87-42fd-93f1-64d112af1ede"). InnerVolumeSpecName "kube-api-access-wdr5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:49:46.265383 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.265314 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-token" (OuterVolumeSpecName: "istio-token") pod "bbd62e91-ec87-42fd-93f1-64d112af1ede" (UID: "bbd62e91-ec87-42fd-93f1-64d112af1ede"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:49:46.363686 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.363650 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-podinfo\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:49:46.363686 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.363678 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-envoy\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:49:46.363686 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.363689 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd62e91-ec87-42fd-93f1-64d112af1ede-istio-token\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:49:46.363926 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.363697 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdr5g\" (UniqueName: \"kubernetes.io/projected/bbd62e91-ec87-42fd-93f1-64d112af1ede-kube-api-access-wdr5g\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:49:46.419818 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.419786 2577 generic.go:356] "Generic (PLEG): container finished" podID="bbd62e91-ec87-42fd-93f1-64d112af1ede" containerID="0fae84923e17a1479c91aa1b21cba9580fadb2c67634b191b3b88eabad799639" exitCode=0 Sep 08 00:49:46.420026 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.419830 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" Sep 08 00:49:46.420026 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.419881 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" event={"ID":"bbd62e91-ec87-42fd-93f1-64d112af1ede","Type":"ContainerDied","Data":"0fae84923e17a1479c91aa1b21cba9580fadb2c67634b191b3b88eabad799639"} Sep 08 00:49:46.420026 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.419933 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n" event={"ID":"bbd62e91-ec87-42fd-93f1-64d112af1ede","Type":"ContainerDied","Data":"23524525f739ade6761bb1c47cdccbaa14761620f80af7ac82526181ff35dabe"} Sep 08 00:49:46.420026 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.419964 2577 scope.go:117] "RemoveContainer" containerID="0fae84923e17a1479c91aa1b21cba9580fadb2c67634b191b3b88eabad799639" Sep 08 00:49:46.429896 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.429878 2577 scope.go:117] "RemoveContainer" containerID="0fae84923e17a1479c91aa1b21cba9580fadb2c67634b191b3b88eabad799639" Sep 08 00:49:46.430226 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:49:46.430200 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fae84923e17a1479c91aa1b21cba9580fadb2c67634b191b3b88eabad799639\": container with ID starting with 0fae84923e17a1479c91aa1b21cba9580fadb2c67634b191b3b88eabad799639 not found: ID does not exist" containerID="0fae84923e17a1479c91aa1b21cba9580fadb2c67634b191b3b88eabad799639" Sep 08 00:49:46.430318 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.430239 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fae84923e17a1479c91aa1b21cba9580fadb2c67634b191b3b88eabad799639"} err="failed to get container status \"0fae84923e17a1479c91aa1b21cba9580fadb2c67634b191b3b88eabad799639\": rpc error: code = NotFound desc = could not find container \"0fae84923e17a1479c91aa1b21cba9580fadb2c67634b191b3b88eabad799639\": container with ID starting with 0fae84923e17a1479c91aa1b21cba9580fadb2c67634b191b3b88eabad799639 not found: ID does not exist" Sep 08 00:49:46.445462 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.445416 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n"] Sep 08 00:49:46.447113 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.447088 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-75c97b-openshift-default-66689f74b-gnj5n"] Sep 08 00:49:46.549218 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.549194 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1cf84f-x8bsq" Sep 08 00:49:46.565179 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.565150 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/335cee51-973f-4378-b212-28e555e71523-maas-api-service-ca\") pod \"335cee51-973f-4378-b212-28e555e71523\" (UID: \"335cee51-973f-4378-b212-28e555e71523\") " Sep 08 00:49:46.565587 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.565562 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335cee51-973f-4378-b212-28e555e71523-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "335cee51-973f-4378-b212-28e555e71523" (UID: "335cee51-973f-4378-b212-28e555e71523"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:49:46.666791 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.666759 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/335cee51-973f-4378-b212-28e555e71523-maas-api-service-ca\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:49:46.791278 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:46.791197 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbd62e91-ec87-42fd-93f1-64d112af1ede" path="/var/lib/kubelet/pods/bbd62e91-ec87-42fd-93f1-64d112af1ede/volumes" Sep 08 00:49:47.093320 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:47.093240 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-5872-139882422240720"] Sep 08 00:49:47.093660 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:47.093555 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbd62e91-ec87-42fd-93f1-64d112af1ede" containerName="istio-proxy" Sep 08 00:49:47.093660 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:47.093567 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbd62e91-ec87-42fd-93f1-64d112af1ede" containerName="istio-proxy" Sep 08 00:49:47.093660 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:47.093574 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="335cee51-973f-4378-b212-28e555e71523" containerName="revoke-keys" Sep 08 00:49:47.093660 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:47.093579 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="335cee51-973f-4378-b212-28e555e71523" containerName="revoke-keys" Sep 08 00:49:47.093660 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:47.093626 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="335cee51-973f-4378-b212-28e555e71523" containerName="revoke-keys" Sep 08 00:49:47.093660 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:47.093636 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbd62e91-ec87-42fd-93f1-64d112af1ede" containerName="istio-proxy" Sep 08 00:49:47.096618 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:47.096601 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5872-139882422240720" Sep 08 00:49:47.100883 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:47.100858 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 08 00:49:47.101049 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:47.100952 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fj6d7\"" Sep 08 00:49:47.101813 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:47.101792 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 08 00:49:47.109827 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:47.109807 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-5872-139882422240720"] Sep 08 00:49:47.171081 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:47.171053 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktwgc\" (UniqueName: \"kubernetes.io/projected/4a8105ab-a9be-4b89-86e2-d938d65c75b9-kube-api-access-ktwgc\") pod \"test-curl-5872-139882422240720\" (UID: \"4a8105ab-a9be-4b89-86e2-d938d65c75b9\") " pod="opendatahub/test-curl-5872-139882422240720" Sep 08 00:49:47.272478 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:47.272440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktwgc\" (UniqueName: \"kubernetes.io/projected/4a8105ab-a9be-4b89-86e2-d938d65c75b9-kube-api-access-ktwgc\") pod \"test-curl-5872-139882422240720\" (UID: \"4a8105ab-a9be-4b89-86e2-d938d65c75b9\") " pod="opendatahub/test-curl-5872-139882422240720" Sep 08 00:49:47.292440 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:47.292414 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktwgc\" (UniqueName: \"kubernetes.io/projected/4a8105ab-a9be-4b89-86e2-d938d65c75b9-kube-api-access-ktwgc\") pod \"test-curl-5872-139882422240720\" (UID: \"4a8105ab-a9be-4b89-86e2-d938d65c75b9\") " pod="opendatahub/test-curl-5872-139882422240720" Sep 08 00:49:47.405385 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:47.405299 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5872-139882422240720" Sep 08 00:49:47.427635 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:47.427611 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1cf84f-x8bsq" Sep 08 00:49:47.427751 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:47.427632 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1cf84f-x8bsq" event={"ID":"335cee51-973f-4378-b212-28e555e71523","Type":"ContainerDied","Data":"e3950ceee8ab9c36bae88b0148b15dda905cab4e9e6ea5ecc8327604591204e1"} Sep 08 00:49:47.427751 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:47.427677 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3950ceee8ab9c36bae88b0148b15dda905cab4e9e6ea5ecc8327604591204e1" Sep 08 00:49:47.672899 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:47.672875 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-5872-139882422240720"] Sep 08 00:49:47.674282 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:49:47.674260 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a8105ab_a9be_4b89_86e2_d938d65c75b9.slice/crio-cdea47fa27ec7a9c93acce844abd86330af1d0ac6d32163d2db2d082a5cbc180 WatchSource:0}: Error finding container cdea47fa27ec7a9c93acce844abd86330af1d0ac6d32163d2db2d082a5cbc180: Status 404 returned error can't find the container with id cdea47fa27ec7a9c93acce844abd86330af1d0ac6d32163d2db2d082a5cbc180 Sep 08 00:49:47.675641 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:47.675626 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 00:49:48.434642 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:48.434606 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-5872-139882422240720" event={"ID":"4a8105ab-a9be-4b89-86e2-d938d65c75b9","Type":"ContainerStarted","Data":"cdea47fa27ec7a9c93acce844abd86330af1d0ac6d32163d2db2d082a5cbc180"} Sep 08 00:49:49.441106 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:49.441065 2577 generic.go:356] "Generic (PLEG): container finished" podID="4a8105ab-a9be-4b89-86e2-d938d65c75b9" containerID="f664e3775aff0bd1b3588ba5aabac7b6523c81f3b12c1720417a6a9b635274ab" exitCode=0 Sep 08 00:49:49.441499 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:49.441125 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-5872-139882422240720" event={"ID":"4a8105ab-a9be-4b89-86e2-d938d65c75b9","Type":"ContainerDied","Data":"f664e3775aff0bd1b3588ba5aabac7b6523c81f3b12c1720417a6a9b635274ab"} Sep 08 00:49:50.598940 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:50.598913 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5872-139882422240720" Sep 08 00:49:50.630825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:50.630798 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-5872-139882422240720_4a8105ab-a9be-4b89-86e2-d938d65c75b9/test-curl-5872-139882422240720/0.log" Sep 08 00:49:50.651642 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:50.651619 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-5872-139882422240720"] Sep 08 00:49:50.655066 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:50.655036 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-5872-139882422240720"] Sep 08 00:49:50.702266 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:50.702205 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktwgc\" (UniqueName: \"kubernetes.io/projected/4a8105ab-a9be-4b89-86e2-d938d65c75b9-kube-api-access-ktwgc\") pod \"4a8105ab-a9be-4b89-86e2-d938d65c75b9\" (UID: \"4a8105ab-a9be-4b89-86e2-d938d65c75b9\") " Sep 08 00:49:50.704364 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:50.704342 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a8105ab-a9be-4b89-86e2-d938d65c75b9-kube-api-access-ktwgc" (OuterVolumeSpecName: "kube-api-access-ktwgc") pod "4a8105ab-a9be-4b89-86e2-d938d65c75b9" (UID: "4a8105ab-a9be-4b89-86e2-d938d65c75b9"). InnerVolumeSpecName "kube-api-access-ktwgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:49:50.765730 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:50.765699 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-5872-139882422241296"] Sep 08 00:49:50.766041 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:50.766027 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a8105ab-a9be-4b89-86e2-d938d65c75b9" containerName="test-curl-5872-139882422240720" Sep 08 00:49:50.766089 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:50.766042 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a8105ab-a9be-4b89-86e2-d938d65c75b9" containerName="test-curl-5872-139882422240720" Sep 08 00:49:50.766125 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:50.766109 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a8105ab-a9be-4b89-86e2-d938d65c75b9" containerName="test-curl-5872-139882422240720" Sep 08 00:49:50.768905 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:50.768890 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5872-139882422241296" Sep 08 00:49:50.772669 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:50.772645 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-5872-139882422241296"] Sep 08 00:49:50.798256 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:50.796631 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a8105ab-a9be-4b89-86e2-d938d65c75b9" path="/var/lib/kubelet/pods/4a8105ab-a9be-4b89-86e2-d938d65c75b9/volumes" Sep 08 00:49:50.802912 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:50.802879 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv866\" (UniqueName: \"kubernetes.io/projected/6d55dfdd-721b-4c6e-a658-f65401277048-kube-api-access-jv866\") pod \"test-curl-5872-139882422241296\" (UID: \"6d55dfdd-721b-4c6e-a658-f65401277048\") " pod="opendatahub/test-curl-5872-139882422241296" Sep 08 00:49:50.803094 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:50.802953 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktwgc\" (UniqueName: \"kubernetes.io/projected/4a8105ab-a9be-4b89-86e2-d938d65c75b9-kube-api-access-ktwgc\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:49:50.904312 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:50.904272 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv866\" (UniqueName: \"kubernetes.io/projected/6d55dfdd-721b-4c6e-a658-f65401277048-kube-api-access-jv866\") pod \"test-curl-5872-139882422241296\" (UID: \"6d55dfdd-721b-4c6e-a658-f65401277048\") " pod="opendatahub/test-curl-5872-139882422241296" Sep 08 00:49:50.912903 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:50.912865 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv866\" (UniqueName: \"kubernetes.io/projected/6d55dfdd-721b-4c6e-a658-f65401277048-kube-api-access-jv866\") pod \"test-curl-5872-139882422241296\" (UID: \"6d55dfdd-721b-4c6e-a658-f65401277048\") " pod="opendatahub/test-curl-5872-139882422241296" Sep 08 00:49:51.086479 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:51.086449 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5872-139882422241296" Sep 08 00:49:51.356745 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:51.356720 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-5872-139882422241296"] Sep 08 00:49:51.358901 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:49:51.358878 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d55dfdd_721b_4c6e_a658_f65401277048.slice/crio-6131583a99bfd01ba67390150e5bd919e9617d0ddeb0297c7d267a79f60cce6b WatchSource:0}: Error finding container 6131583a99bfd01ba67390150e5bd919e9617d0ddeb0297c7d267a79f60cce6b: Status 404 returned error can't find the container with id 6131583a99bfd01ba67390150e5bd919e9617d0ddeb0297c7d267a79f60cce6b Sep 08 00:49:51.453901 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:51.453864 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-5872-139882422241296" event={"ID":"6d55dfdd-721b-4c6e-a658-f65401277048","Type":"ContainerStarted","Data":"6131583a99bfd01ba67390150e5bd919e9617d0ddeb0297c7d267a79f60cce6b"} Sep 08 00:49:51.455055 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:51.455033 2577 scope.go:117] "RemoveContainer" containerID="f664e3775aff0bd1b3588ba5aabac7b6523c81f3b12c1720417a6a9b635274ab" Sep 08 00:49:51.455055 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:51.455051 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5872-139882422240720" Sep 08 00:49:52.462098 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:52.461982 2577 generic.go:356] "Generic (PLEG): container finished" podID="6d55dfdd-721b-4c6e-a658-f65401277048" containerID="44bedde2e658059584ba9a49936814b991327bef482023da6138634ac3af07d0" exitCode=0 Sep 08 00:49:52.462098 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:52.462064 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-5872-139882422241296" event={"ID":"6d55dfdd-721b-4c6e-a658-f65401277048","Type":"ContainerDied","Data":"44bedde2e658059584ba9a49936814b991327bef482023da6138634ac3af07d0"} Sep 08 00:49:53.597687 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:53.597658 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5872-139882422241296" Sep 08 00:49:53.617614 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:53.617582 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-5872-139882422241296_6d55dfdd-721b-4c6e-a658-f65401277048/test-curl-5872-139882422241296/0.log" Sep 08 00:49:53.629910 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:53.629889 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jv866\" (UniqueName: \"kubernetes.io/projected/6d55dfdd-721b-4c6e-a658-f65401277048-kube-api-access-jv866\") pod \"6d55dfdd-721b-4c6e-a658-f65401277048\" (UID: \"6d55dfdd-721b-4c6e-a658-f65401277048\") " Sep 08 00:49:53.631851 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:53.631824 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d55dfdd-721b-4c6e-a658-f65401277048-kube-api-access-jv866" (OuterVolumeSpecName: "kube-api-access-jv866") pod "6d55dfdd-721b-4c6e-a658-f65401277048" (UID: "6d55dfdd-721b-4c6e-a658-f65401277048"). InnerVolumeSpecName "kube-api-access-jv866". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:49:53.639593 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:53.639572 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-5872-139882422241296"] Sep 08 00:49:53.645419 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:53.645397 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-5872-139882422241296"] Sep 08 00:49:53.730861 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:53.730776 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv866\" (UniqueName: \"kubernetes.io/projected/6d55dfdd-721b-4c6e-a658-f65401277048-kube-api-access-jv866\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:49:53.752554 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:53.752523 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-5872-139882422241296"] Sep 08 00:49:53.752844 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:53.752831 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d55dfdd-721b-4c6e-a658-f65401277048" containerName="test-curl-5872-139882422241296" Sep 08 00:49:53.752896 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:53.752845 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d55dfdd-721b-4c6e-a658-f65401277048" containerName="test-curl-5872-139882422241296" Sep 08 00:49:53.752940 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:53.752908 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d55dfdd-721b-4c6e-a658-f65401277048" containerName="test-curl-5872-139882422241296" Sep 08 00:49:53.755690 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:53.755675 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5872-139882422241296" Sep 08 00:49:53.762270 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:53.762250 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-5872-139882422241296"] Sep 08 00:49:53.832188 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:53.832152 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxhnb\" (UniqueName: \"kubernetes.io/projected/fe32ba23-1c35-406e-92f1-e6bcac12d320-kube-api-access-cxhnb\") pod \"test-curl-5872-139882422241296\" (UID: \"fe32ba23-1c35-406e-92f1-e6bcac12d320\") " pod="opendatahub/test-curl-5872-139882422241296" Sep 08 00:49:53.932866 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:53.932830 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxhnb\" (UniqueName: \"kubernetes.io/projected/fe32ba23-1c35-406e-92f1-e6bcac12d320-kube-api-access-cxhnb\") pod \"test-curl-5872-139882422241296\" (UID: \"fe32ba23-1c35-406e-92f1-e6bcac12d320\") " pod="opendatahub/test-curl-5872-139882422241296" Sep 08 00:49:53.940837 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:53.940815 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxhnb\" (UniqueName: \"kubernetes.io/projected/fe32ba23-1c35-406e-92f1-e6bcac12d320-kube-api-access-cxhnb\") pod \"test-curl-5872-139882422241296\" (UID: \"fe32ba23-1c35-406e-92f1-e6bcac12d320\") " pod="opendatahub/test-curl-5872-139882422241296" Sep 08 00:49:54.066233 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:54.066202 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5872-139882422241296" Sep 08 00:49:54.333025 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:54.332967 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-5872-139882422241296"] Sep 08 00:49:54.334634 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:49:54.334608 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe32ba23_1c35_406e_92f1_e6bcac12d320.slice/crio-1561c45d1a425c9908802e4f3ee5e69b8bffbe04751834cd876409d81fcdfa92 WatchSource:0}: Error finding container 1561c45d1a425c9908802e4f3ee5e69b8bffbe04751834cd876409d81fcdfa92: Status 404 returned error can't find the container with id 1561c45d1a425c9908802e4f3ee5e69b8bffbe04751834cd876409d81fcdfa92 Sep 08 00:49:54.427737 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:54.427705 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1cf84f-29813805-dcmdg"] Sep 08 00:49:54.430972 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:54.430946 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1cf84f-29813805-dcmdg"] Sep 08 00:49:54.476948 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:54.476908 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-5872-139882422241296" event={"ID":"fe32ba23-1c35-406e-92f1-e6bcac12d320","Type":"ContainerStarted","Data":"1561c45d1a425c9908802e4f3ee5e69b8bffbe04751834cd876409d81fcdfa92"} Sep 08 00:49:54.478709 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:54.478525 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6131583a99bfd01ba67390150e5bd919e9617d0ddeb0297c7d267a79f60cce6b" Sep 08 00:49:54.478709 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:54.478597 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5872-139882422241296" Sep 08 00:49:54.491072 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:54.491035 2577 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-5872-139882422241296" oldPodUID="6d55dfdd-721b-4c6e-a658-f65401277048" podUID="fe32ba23-1c35-406e-92f1-e6bcac12d320" Sep 08 00:49:54.621632 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:54.621540 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1cf84f-x8bsq"] Sep 08 00:49:54.623800 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:54.623774 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1cf84f-x8bsq"] Sep 08 00:49:54.790116 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:54.790086 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01d6e639-0bb4-4032-94f4-6c039b71a3d1" path="/var/lib/kubelet/pods/01d6e639-0bb4-4032-94f4-6c039b71a3d1/volumes" Sep 08 00:49:54.790408 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:54.790395 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="335cee51-973f-4378-b212-28e555e71523" path="/var/lib/kubelet/pods/335cee51-973f-4378-b212-28e555e71523/volumes" Sep 08 00:49:54.790675 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:54.790664 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d55dfdd-721b-4c6e-a658-f65401277048" path="/var/lib/kubelet/pods/6d55dfdd-721b-4c6e-a658-f65401277048/volumes" Sep 08 00:49:55.487378 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:55.487336 2577 generic.go:356] "Generic (PLEG): container finished" podID="fe32ba23-1c35-406e-92f1-e6bcac12d320" containerID="12fd99ac5aceee6e3624b9c7f8b9c7810d5d8a46c23c4fa8dbd1c1f14f122463" exitCode=0 Sep 08 00:49:55.487593 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:55.487422 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-5872-139882422241296" event={"ID":"fe32ba23-1c35-406e-92f1-e6bcac12d320","Type":"ContainerDied","Data":"12fd99ac5aceee6e3624b9c7f8b9c7810d5d8a46c23c4fa8dbd1c1f14f122463"} Sep 08 00:49:56.621228 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:56.621202 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5872-139882422241296" Sep 08 00:49:56.640092 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:56.640063 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-5872-139882422241296_fe32ba23-1c35-406e-92f1-e6bcac12d320/test-curl-5872-139882422241296/0.log" Sep 08 00:49:56.660638 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:56.660611 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-5872-139882422241296"] Sep 08 00:49:56.664099 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:56.664073 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-5872-139882422241296"] Sep 08 00:49:56.664271 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:56.664251 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxhnb\" (UniqueName: \"kubernetes.io/projected/fe32ba23-1c35-406e-92f1-e6bcac12d320-kube-api-access-cxhnb\") pod \"fe32ba23-1c35-406e-92f1-e6bcac12d320\" (UID: \"fe32ba23-1c35-406e-92f1-e6bcac12d320\") " Sep 08 00:49:56.666201 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:56.666174 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe32ba23-1c35-406e-92f1-e6bcac12d320-kube-api-access-cxhnb" (OuterVolumeSpecName: "kube-api-access-cxhnb") pod "fe32ba23-1c35-406e-92f1-e6bcac12d320" (UID: "fe32ba23-1c35-406e-92f1-e6bcac12d320"). InnerVolumeSpecName "kube-api-access-cxhnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:49:56.765716 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:56.765629 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxhnb\" (UniqueName: \"kubernetes.io/projected/fe32ba23-1c35-406e-92f1-e6bcac12d320-kube-api-access-cxhnb\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:49:56.791135 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:56.791103 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe32ba23-1c35-406e-92f1-e6bcac12d320" path="/var/lib/kubelet/pods/fe32ba23-1c35-406e-92f1-e6bcac12d320/volumes" Sep 08 00:49:57.502587 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.502557 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-5872-139882422241296" Sep 08 00:49:57.502785 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.502597 2577 scope.go:117] "RemoveContainer" containerID="12fd99ac5aceee6e3624b9c7f8b9c7810d5d8a46c23c4fa8dbd1c1f14f122463" Sep 08 00:49:57.504287 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.503890 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5"] Sep 08 00:49:57.504379 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.504368 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe32ba23-1c35-406e-92f1-e6bcac12d320" containerName="test-curl-5872-139882422241296" Sep 08 00:49:57.504445 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.504386 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe32ba23-1c35-406e-92f1-e6bcac12d320" containerName="test-curl-5872-139882422241296" Sep 08 00:49:57.504501 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.504481 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe32ba23-1c35-406e-92f1-e6bcac12d320" containerName="test-curl-5872-139882422241296" Sep 08 00:49:57.507739 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.507719 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.510290 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.510265 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c28b2155-openshift-default-dockercfg-jk7nr\"" Sep 08 00:49:57.519531 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.519509 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5"] Sep 08 00:49:57.574461 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.574426 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-token\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.574614 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.574475 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-podinfo\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.574614 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.574499 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-data\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.574705 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.574592 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-workload-certs\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.574705 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.574663 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-envoy\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.574770 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.574711 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqhmc\" (UniqueName: \"kubernetes.io/projected/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-kube-api-access-wqhmc\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.574770 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.574733 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istiod-ca-cert\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.574770 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.574766 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-credential-socket\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.574881 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.574788 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-workload-socket\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.675772 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.675733 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-credential-socket\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.675772 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.675773 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-workload-socket\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.676300 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.675792 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-token\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.676300 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.675823 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-podinfo\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.676300 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.675841 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-data\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.676300 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.675865 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-workload-certs\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.676300 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.675888 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-envoy\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.676300 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.675926 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqhmc\" (UniqueName: \"kubernetes.io/projected/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-kube-api-access-wqhmc\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.676300 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.675954 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istiod-ca-cert\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.676300 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.676236 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-credential-socket\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.676300 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.676272 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-workload-socket\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.676700 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.676440 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-data\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.676700 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.676475 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-workload-certs\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.676700 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.676658 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istiod-ca-cert\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.678262 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.678239 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-envoy\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.678407 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.678387 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-podinfo\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.683719 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.683691 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-token\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.684084 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.684063 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqhmc\" (UniqueName: \"kubernetes.io/projected/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-kube-api-access-wqhmc\") pod \"e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:57.819788 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:57.819754 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:58.102225 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:58.102195 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5"] Sep 08 00:49:58.103976 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:49:58.103955 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd1765e_c6fd_4b46_8748_ccaf159d7f84.slice/crio-04304beba3da41cd4366e6be0a5ff800a7e790ec8fcb5c1c61da56b10cf583fb WatchSource:0}: Error finding container 04304beba3da41cd4366e6be0a5ff800a7e790ec8fcb5c1c61da56b10cf583fb: Status 404 returned error can't find the container with id 04304beba3da41cd4366e6be0a5ff800a7e790ec8fcb5c1c61da56b10cf583fb Sep 08 00:49:58.106439 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:58.106397 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:49:58.106514 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:58.106486 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:49:58.106558 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:58.106530 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:49:58.509656 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:58.509556 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" event={"ID":"0bd1765e-c6fd-4b46-8748-ccaf159d7f84","Type":"ContainerStarted","Data":"e6dda66b30c94f7157738b93433cbd0167e6b9de78ef23ce4dfd24e856e1df32"} Sep 08 00:49:58.509656 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:58.509609 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" event={"ID":"0bd1765e-c6fd-4b46-8748-ccaf159d7f84","Type":"ContainerStarted","Data":"04304beba3da41cd4366e6be0a5ff800a7e790ec8fcb5c1c61da56b10cf583fb"} Sep 08 00:49:58.531593 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:58.531547 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" podStartSLOduration=1.531532093 podStartE2EDuration="1.531532093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:49:58.52965242 +0000 UTC m=+1012.224893582" watchObservedRunningTime="2026-09-08 00:49:58.531532093 +0000 UTC m=+1012.226773252" Sep 08 00:49:58.820394 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:58.820359 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:49:59.825312 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:49:59.825281 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:50:00.521701 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:00.521668 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:50:00.522626 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:00.522607 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:50:01.987132 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:01.987097 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3896b-5q965"] Sep 08 00:50:01.990098 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:01.990078 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3896b-5q965" Sep 08 00:50:01.992554 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:01.992527 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dqw2\"" Sep 08 00:50:01.992554 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:01.992534 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 00:50:01.998012 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:01.997902 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3896b-5q965"] Sep 08 00:50:02.117916 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:02.117884 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7744d3-8d98-4826-9d5c-be7875ca7933-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e3896b-5q965\" (UID: \"6a7744d3-8d98-4826-9d5c-be7875ca7933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3896b-5q965" Sep 08 00:50:02.219200 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:02.219164 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7744d3-8d98-4826-9d5c-be7875ca7933-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e3896b-5q965\" (UID: \"6a7744d3-8d98-4826-9d5c-be7875ca7933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3896b-5q965" Sep 08 00:50:02.219802 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:02.219783 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7744d3-8d98-4826-9d5c-be7875ca7933-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-e3896b-5q965\" (UID: \"6a7744d3-8d98-4826-9d5c-be7875ca7933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3896b-5q965" Sep 08 00:50:02.301512 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:02.301468 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3896b-5q965" Sep 08 00:50:02.580374 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:02.580348 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3896b-5q965"] Sep 08 00:50:02.582871 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:50:02.582839 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7744d3_8d98_4826_9d5c_be7875ca7933.slice/crio-2233d751aec0c47b02fd629f34b68da0d244cb395e275d256c115b07ca35a273 WatchSource:0}: Error finding container 2233d751aec0c47b02fd629f34b68da0d244cb395e275d256c115b07ca35a273: Status 404 returned error can't find the container with id 2233d751aec0c47b02fd629f34b68da0d244cb395e275d256c115b07ca35a273 Sep 08 00:50:03.538755 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:03.538719 2577 generic.go:356] "Generic (PLEG): container finished" podID="6a7744d3-8d98-4826-9d5c-be7875ca7933" containerID="d47269076ab678da5a53be97b9b171ee84f5f462f4b15115bbd16a5f622fe1e9" exitCode=0 Sep 08 00:50:03.539141 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:03.538786 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3896b-5q965" event={"ID":"6a7744d3-8d98-4826-9d5c-be7875ca7933","Type":"ContainerDied","Data":"d47269076ab678da5a53be97b9b171ee84f5f462f4b15115bbd16a5f622fe1e9"} Sep 08 00:50:03.539141 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:03.538848 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3896b-5q965" event={"ID":"6a7744d3-8d98-4826-9d5c-be7875ca7933","Type":"ContainerStarted","Data":"2233d751aec0c47b02fd629f34b68da0d244cb395e275d256c115b07ca35a273"} Sep 08 00:50:04.671312 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:04.671287 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3896b-5q965" Sep 08 00:50:04.740574 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:04.740544 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7744d3-8d98-4826-9d5c-be7875ca7933-maas-api-service-ca\") pod \"6a7744d3-8d98-4826-9d5c-be7875ca7933\" (UID: \"6a7744d3-8d98-4826-9d5c-be7875ca7933\") " Sep 08 00:50:04.740890 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:04.740867 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7744d3-8d98-4826-9d5c-be7875ca7933-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a7744d3-8d98-4826-9d5c-be7875ca7933" (UID: "6a7744d3-8d98-4826-9d5c-be7875ca7933"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:50:04.841603 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:04.841561 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7744d3-8d98-4826-9d5c-be7875ca7933-maas-api-service-ca\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:50:05.550064 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:05.550035 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3896b-5q965" Sep 08 00:50:05.550272 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:05.550029 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3896b-5q965" event={"ID":"6a7744d3-8d98-4826-9d5c-be7875ca7933","Type":"ContainerDied","Data":"2233d751aec0c47b02fd629f34b68da0d244cb395e275d256c115b07ca35a273"} Sep 08 00:50:05.550272 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:05.550156 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2233d751aec0c47b02fd629f34b68da0d244cb395e275d256c115b07ca35a273" Sep 08 00:50:06.801520 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:06.801481 2577 scope.go:117] "RemoveContainer" containerID="4d3ccdf0ac7035fa6ce47cfd76095e37074d4c89f803c002595d8b3c81422b52" Sep 08 00:50:12.744672 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:12.744635 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b"] Sep 08 00:50:12.745090 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:12.744918 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b" podUID="c065d6b7-06a5-4f9a-ba41-9cfb027bfa40" containerName="payload-processing" containerID="cri-o://b8cf3267fdd22bbcab221364aaa47fa4a95384aabc34cd15e5f5376bfe4911a5" gracePeriod=30 Sep 08 00:50:12.751237 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:12.751205 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h"] Sep 08 00:50:12.751493 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:12.751446 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h" podUID="6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb" containerName="payload-pre-processing" containerID="cri-o://2a458f7324694c1a108155aaf54db279124327c78a77d3729ec7e22a0e705b29" gracePeriod=30 Sep 08 00:50:12.902812 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:12.902776 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3896b-5q965"] Sep 08 00:50:12.906621 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:12.906590 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-e3896b-5q965"] Sep 08 00:50:13.002669 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.002598 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b" Sep 08 00:50:13.038790 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.038764 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h" Sep 08 00:50:13.113139 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.113101 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q2sq\" (UniqueName: \"kubernetes.io/projected/6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb-kube-api-access-6q2sq\") pod \"6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb\" (UID: \"6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb\") " Sep 08 00:50:13.113313 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.113169 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c065d6b7-06a5-4f9a-ba41-9cfb027bfa40-plugins-config-volume\") pod \"c065d6b7-06a5-4f9a-ba41-9cfb027bfa40\" (UID: \"c065d6b7-06a5-4f9a-ba41-9cfb027bfa40\") " Sep 08 00:50:13.113313 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.113228 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mf55k\" (UniqueName: \"kubernetes.io/projected/c065d6b7-06a5-4f9a-ba41-9cfb027bfa40-kube-api-access-mf55k\") pod \"c065d6b7-06a5-4f9a-ba41-9cfb027bfa40\" (UID: \"c065d6b7-06a5-4f9a-ba41-9cfb027bfa40\") " Sep 08 00:50:13.113627 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.113596 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c065d6b7-06a5-4f9a-ba41-9cfb027bfa40-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c065d6b7-06a5-4f9a-ba41-9cfb027bfa40" (UID: "c065d6b7-06a5-4f9a-ba41-9cfb027bfa40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:50:13.115263 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.115242 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c065d6b7-06a5-4f9a-ba41-9cfb027bfa40-kube-api-access-mf55k" (OuterVolumeSpecName: "kube-api-access-mf55k") pod "c065d6b7-06a5-4f9a-ba41-9cfb027bfa40" (UID: "c065d6b7-06a5-4f9a-ba41-9cfb027bfa40"). InnerVolumeSpecName "kube-api-access-mf55k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:50:13.115362 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.115329 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb-kube-api-access-6q2sq" (OuterVolumeSpecName: "kube-api-access-6q2sq") pod "6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb" (UID: "6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb"). InnerVolumeSpecName "kube-api-access-6q2sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:50:13.213947 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.213912 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb-plugins-config-volume\") pod \"6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb\" (UID: \"6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb\") " Sep 08 00:50:13.214256 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.214239 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q2sq\" (UniqueName: \"kubernetes.io/projected/6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb-kube-api-access-6q2sq\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:50:13.214333 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.214262 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c065d6b7-06a5-4f9a-ba41-9cfb027bfa40-plugins-config-volume\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:50:13.214333 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.214259 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb" (UID: "6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:50:13.214333 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.214275 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf55k\" (UniqueName: \"kubernetes.io/projected/c065d6b7-06a5-4f9a-ba41-9cfb027bfa40-kube-api-access-mf55k\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:50:13.315273 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.315239 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb-plugins-config-volume\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:50:13.591541 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.591445 2577 generic.go:356] "Generic (PLEG): container finished" podID="6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb" containerID="2a458f7324694c1a108155aaf54db279124327c78a77d3729ec7e22a0e705b29" exitCode=0 Sep 08 00:50:13.591541 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.591493 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h" Sep 08 00:50:13.591780 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.591532 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h" event={"ID":"6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb","Type":"ContainerDied","Data":"2a458f7324694c1a108155aaf54db279124327c78a77d3729ec7e22a0e705b29"} Sep 08 00:50:13.591780 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.591591 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h" event={"ID":"6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb","Type":"ContainerDied","Data":"56066bb9112f175284e17f1c90d46aaed30e188c3f1570083fd3bd213f4d0220"} Sep 08 00:50:13.591780 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.591626 2577 scope.go:117] "RemoveContainer" containerID="2a458f7324694c1a108155aaf54db279124327c78a77d3729ec7e22a0e705b29" Sep 08 00:50:13.593070 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.593047 2577 generic.go:356] "Generic (PLEG): container finished" podID="c065d6b7-06a5-4f9a-ba41-9cfb027bfa40" containerID="b8cf3267fdd22bbcab221364aaa47fa4a95384aabc34cd15e5f5376bfe4911a5" exitCode=0 Sep 08 00:50:13.593188 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.593109 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b" event={"ID":"c065d6b7-06a5-4f9a-ba41-9cfb027bfa40","Type":"ContainerDied","Data":"b8cf3267fdd22bbcab221364aaa47fa4a95384aabc34cd15e5f5376bfe4911a5"} Sep 08 00:50:13.593188 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.593151 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b" event={"ID":"c065d6b7-06a5-4f9a-ba41-9cfb027bfa40","Type":"ContainerDied","Data":"0c33f50da2fe9106b58b489ef03b5f6d9d9eb0e8ce442982dc38486a8b24dd73"} Sep 08 00:50:13.593188 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.593115 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b" Sep 08 00:50:13.600786 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.600768 2577 scope.go:117] "RemoveContainer" containerID="2a458f7324694c1a108155aaf54db279124327c78a77d3729ec7e22a0e705b29" Sep 08 00:50:13.601078 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:50:13.601056 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a458f7324694c1a108155aaf54db279124327c78a77d3729ec7e22a0e705b29\": container with ID starting with 2a458f7324694c1a108155aaf54db279124327c78a77d3729ec7e22a0e705b29 not found: ID does not exist" containerID="2a458f7324694c1a108155aaf54db279124327c78a77d3729ec7e22a0e705b29" Sep 08 00:50:13.601168 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.601089 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a458f7324694c1a108155aaf54db279124327c78a77d3729ec7e22a0e705b29"} err="failed to get container status \"2a458f7324694c1a108155aaf54db279124327c78a77d3729ec7e22a0e705b29\": rpc error: code = NotFound desc = could not find container \"2a458f7324694c1a108155aaf54db279124327c78a77d3729ec7e22a0e705b29\": container with ID starting with 2a458f7324694c1a108155aaf54db279124327c78a77d3729ec7e22a0e705b29 not found: ID does not exist" Sep 08 00:50:13.601168 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.601111 2577 scope.go:117] "RemoveContainer" containerID="b8cf3267fdd22bbcab221364aaa47fa4a95384aabc34cd15e5f5376bfe4911a5" Sep 08 00:50:13.609103 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.609085 2577 scope.go:117] "RemoveContainer" containerID="b8cf3267fdd22bbcab221364aaa47fa4a95384aabc34cd15e5f5376bfe4911a5" Sep 08 00:50:13.609344 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:50:13.609325 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8cf3267fdd22bbcab221364aaa47fa4a95384aabc34cd15e5f5376bfe4911a5\": container with ID starting with b8cf3267fdd22bbcab221364aaa47fa4a95384aabc34cd15e5f5376bfe4911a5 not found: ID does not exist" containerID="b8cf3267fdd22bbcab221364aaa47fa4a95384aabc34cd15e5f5376bfe4911a5" Sep 08 00:50:13.609393 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.609350 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8cf3267fdd22bbcab221364aaa47fa4a95384aabc34cd15e5f5376bfe4911a5"} err="failed to get container status \"b8cf3267fdd22bbcab221364aaa47fa4a95384aabc34cd15e5f5376bfe4911a5\": rpc error: code = NotFound desc = could not find container \"b8cf3267fdd22bbcab221364aaa47fa4a95384aabc34cd15e5f5376bfe4911a5\": container with ID starting with b8cf3267fdd22bbcab221364aaa47fa4a95384aabc34cd15e5f5376bfe4911a5 not found: ID does not exist" Sep 08 00:50:13.616564 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.616544 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b"] Sep 08 00:50:13.622710 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.622686 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e3896b-65d65f877c-8qj2b"] Sep 08 00:50:13.634480 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.634451 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h"] Sep 08 00:50:13.640456 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:13.640438 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e3896b-5fdf6787c8-h9q2h"] Sep 08 00:50:14.789957 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:14.789920 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a7744d3-8d98-4826-9d5c-be7875ca7933" path="/var/lib/kubelet/pods/6a7744d3-8d98-4826-9d5c-be7875ca7933/volumes" Sep 08 00:50:14.790488 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:14.790463 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb" path="/var/lib/kubelet/pods/6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb/volumes" Sep 08 00:50:14.790899 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:14.790876 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c065d6b7-06a5-4f9a-ba41-9cfb027bfa40" path="/var/lib/kubelet/pods/c065d6b7-06a5-4f9a-ba41-9cfb027bfa40/volumes" Sep 08 00:50:19.243379 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:19.243342 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw"] Sep 08 00:50:19.243744 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:19.243562 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" podUID="009b85b2-8630-42b6-875a-ed084b0593b6" containerName="istio-proxy" containerID="cri-o://87e3d4274cd843ccd69491bf24b7eb0e80cd2d500a227a6461d37795a13534cd" gracePeriod=30 Sep 08 00:50:20.037042 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.036533 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck"] Sep 08 00:50:20.037042 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.037043 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c065d6b7-06a5-4f9a-ba41-9cfb027bfa40" containerName="payload-processing" Sep 08 00:50:20.037289 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.037063 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c065d6b7-06a5-4f9a-ba41-9cfb027bfa40" containerName="payload-processing" Sep 08 00:50:20.037289 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.037086 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb" containerName="payload-pre-processing" Sep 08 00:50:20.037289 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.037096 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb" containerName="payload-pre-processing" Sep 08 00:50:20.037289 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.037106 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a7744d3-8d98-4826-9d5c-be7875ca7933" containerName="revoke-keys" Sep 08 00:50:20.037289 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.037115 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a7744d3-8d98-4826-9d5c-be7875ca7933" containerName="revoke-keys" Sep 08 00:50:20.037289 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.037200 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a8aa5a5-8d5e-4f7f-b731-9827e7798bbb" containerName="payload-pre-processing" Sep 08 00:50:20.037289 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.037214 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c065d6b7-06a5-4f9a-ba41-9cfb027bfa40" containerName="payload-processing" Sep 08 00:50:20.037289 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.037226 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a7744d3-8d98-4826-9d5c-be7875ca7933" containerName="revoke-keys" Sep 08 00:50:20.041892 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.041866 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.044393 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.044368 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-b8c4de-openshift-default-dockercfg-f95qd\"" Sep 08 00:50:20.051346 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.051323 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck"] Sep 08 00:50:20.176390 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.176360 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-podinfo\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.176606 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.176398 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhvt7\" (UniqueName: \"kubernetes.io/projected/cb68d759-3712-4438-8ed8-19b3e331c7b5-kube-api-access-zhvt7\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.176606 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.176433 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-workload-socket\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.176606 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.176490 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb68d759-3712-4438-8ed8-19b3e331c7b5-istiod-ca-cert\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.176606 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.176507 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-workload-certs\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.176606 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.176597 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-token\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.176856 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.176636 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-credential-socket\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.176856 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.176703 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-envoy\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.176856 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.176749 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-data\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.277567 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.277527 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-token\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.277567 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.277573 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-credential-socket\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.278119 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.277622 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-envoy\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.278119 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.277647 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-data\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.278119 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.277688 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-podinfo\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.278119 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.277713 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhvt7\" (UniqueName: \"kubernetes.io/projected/cb68d759-3712-4438-8ed8-19b3e331c7b5-kube-api-access-zhvt7\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.278119 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.277745 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-workload-socket\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.278119 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.277783 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb68d759-3712-4438-8ed8-19b3e331c7b5-istiod-ca-cert\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.278119 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.277807 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-workload-certs\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.278502 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.278401 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-workload-socket\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.278502 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.278457 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-data\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.278599 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.278518 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-credential-socket\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.278599 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.278564 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-workload-certs\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.278599 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.278584 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb68d759-3712-4438-8ed8-19b3e331c7b5-istiod-ca-cert\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.279955 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.279935 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-envoy\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.280136 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.280117 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-podinfo\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.285549 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.285499 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-token\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.285704 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.285686 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhvt7\" (UniqueName: \"kubernetes.io/projected/cb68d759-3712-4438-8ed8-19b3e331c7b5-kube-api-access-zhvt7\") pod \"e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.353958 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.353859 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:20.638659 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.638630 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck"] Sep 08 00:50:20.640635 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:50:20.640606 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb68d759_3712_4438_8ed8_19b3e331c7b5.slice/crio-78f3a4643c455b10687b418e2dd9a98c421925950cac9895d2e599a51ecb971d WatchSource:0}: Error finding container 78f3a4643c455b10687b418e2dd9a98c421925950cac9895d2e599a51ecb971d: Status 404 returned error can't find the container with id 78f3a4643c455b10687b418e2dd9a98c421925950cac9895d2e599a51ecb971d Sep 08 00:50:20.643272 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.643235 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:50:20.643377 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.643315 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:50:20.643377 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:20.643361 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:50:21.641919 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:21.641881 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" event={"ID":"cb68d759-3712-4438-8ed8-19b3e331c7b5","Type":"ContainerStarted","Data":"24720273b54c3d0008ee8ecfdbbf7fadd15bc7077ff2a26b2328c34624b6aefc"} Sep 08 00:50:21.642429 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:21.641940 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" event={"ID":"cb68d759-3712-4438-8ed8-19b3e331c7b5","Type":"ContainerStarted","Data":"78f3a4643c455b10687b418e2dd9a98c421925950cac9895d2e599a51ecb971d"} Sep 08 00:50:21.662883 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:21.662827 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" podStartSLOduration=1.6628104559999999 podStartE2EDuration="1.662810456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:50:21.661354048 +0000 UTC m=+1035.356595212" watchObservedRunningTime="2026-09-08 00:50:21.662810456 +0000 UTC m=+1035.358051614" Sep 08 00:50:22.354067 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:22.354024 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:22.359413 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:22.359385 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:22.646960 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:22.646885 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:22.647974 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:22.647957 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:23.652715 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:23.652682 2577 generic.go:356] "Generic (PLEG): container finished" podID="0bd1765e-c6fd-4b46-8748-ccaf159d7f84" containerID="e6dda66b30c94f7157738b93433cbd0167e6b9de78ef23ce4dfd24e856e1df32" exitCode=0 Sep 08 00:50:23.653121 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:23.652756 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" event={"ID":"0bd1765e-c6fd-4b46-8748-ccaf159d7f84","Type":"ContainerDied","Data":"e6dda66b30c94f7157738b93433cbd0167e6b9de78ef23ce4dfd24e856e1df32"} Sep 08 00:50:23.653274 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:23.653253 2577 scope.go:117] "RemoveContainer" containerID="e6dda66b30c94f7157738b93433cbd0167e6b9de78ef23ce4dfd24e856e1df32" Sep 08 00:50:23.654344 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:23.654318 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:50:23.654409 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:23.654380 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:50:23.654446 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:23.654411 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:50:24.523346 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.523319 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:50:24.660788 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.660755 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" event={"ID":"0bd1765e-c6fd-4b46-8748-ccaf159d7f84","Type":"ContainerStarted","Data":"c7dd66addaa88af7786778d2ac76c1f7dcdca45ad0d7a7e74810a26148838557"} Sep 08 00:50:24.662808 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.662784 2577 generic.go:356] "Generic (PLEG): container finished" podID="009b85b2-8630-42b6-875a-ed084b0593b6" containerID="87e3d4274cd843ccd69491bf24b7eb0e80cd2d500a227a6461d37795a13534cd" exitCode=0 Sep 08 00:50:24.662955 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.662835 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" Sep 08 00:50:24.662955 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.662878 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" event={"ID":"009b85b2-8630-42b6-875a-ed084b0593b6","Type":"ContainerDied","Data":"87e3d4274cd843ccd69491bf24b7eb0e80cd2d500a227a6461d37795a13534cd"} Sep 08 00:50:24.662955 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.662928 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw" event={"ID":"009b85b2-8630-42b6-875a-ed084b0593b6","Type":"ContainerDied","Data":"29f0831c28aa32b2f7be9796a3206cb38194de0575894855e4ab660816cef295"} Sep 08 00:50:24.662955 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.662955 2577 scope.go:117] "RemoveContainer" containerID="87e3d4274cd843ccd69491bf24b7eb0e80cd2d500a227a6461d37795a13534cd" Sep 08 00:50:24.671885 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.671867 2577 scope.go:117] "RemoveContainer" containerID="9420baf779f185608d5fd580a71cff614abaa18104011e43385b7a6024514b12" Sep 08 00:50:24.679788 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.679772 2577 scope.go:117] "RemoveContainer" containerID="87e3d4274cd843ccd69491bf24b7eb0e80cd2d500a227a6461d37795a13534cd" Sep 08 00:50:24.680068 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:50:24.680047 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87e3d4274cd843ccd69491bf24b7eb0e80cd2d500a227a6461d37795a13534cd\": container with ID starting with 87e3d4274cd843ccd69491bf24b7eb0e80cd2d500a227a6461d37795a13534cd not found: ID does not exist" containerID="87e3d4274cd843ccd69491bf24b7eb0e80cd2d500a227a6461d37795a13534cd" Sep 08 00:50:24.680119 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.680076 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87e3d4274cd843ccd69491bf24b7eb0e80cd2d500a227a6461d37795a13534cd"} err="failed to get container status \"87e3d4274cd843ccd69491bf24b7eb0e80cd2d500a227a6461d37795a13534cd\": rpc error: code = NotFound desc = could not find container \"87e3d4274cd843ccd69491bf24b7eb0e80cd2d500a227a6461d37795a13534cd\": container with ID starting with 87e3d4274cd843ccd69491bf24b7eb0e80cd2d500a227a6461d37795a13534cd not found: ID does not exist" Sep 08 00:50:24.680119 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.680094 2577 scope.go:117] "RemoveContainer" containerID="9420baf779f185608d5fd580a71cff614abaa18104011e43385b7a6024514b12" Sep 08 00:50:24.680320 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:50:24.680305 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9420baf779f185608d5fd580a71cff614abaa18104011e43385b7a6024514b12\": container with ID starting with 9420baf779f185608d5fd580a71cff614abaa18104011e43385b7a6024514b12 not found: ID does not exist" containerID="9420baf779f185608d5fd580a71cff614abaa18104011e43385b7a6024514b12" Sep 08 00:50:24.680362 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.680326 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9420baf779f185608d5fd580a71cff614abaa18104011e43385b7a6024514b12"} err="failed to get container status \"9420baf779f185608d5fd580a71cff614abaa18104011e43385b7a6024514b12\": rpc error: code = NotFound desc = could not find container \"9420baf779f185608d5fd580a71cff614abaa18104011e43385b7a6024514b12\": container with ID starting with 9420baf779f185608d5fd580a71cff614abaa18104011e43385b7a6024514b12 not found: ID does not exist" Sep 08 00:50:24.721640 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.721602 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-istio-data\") pod \"009b85b2-8630-42b6-875a-ed084b0593b6\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " Sep 08 00:50:24.721640 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.721642 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-istio-envoy\") pod \"009b85b2-8630-42b6-875a-ed084b0593b6\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " Sep 08 00:50:24.721894 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.721698 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-credential-socket\") pod \"009b85b2-8630-42b6-875a-ed084b0593b6\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " Sep 08 00:50:24.721894 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.721739 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009b85b2-8630-42b6-875a-ed084b0593b6-istio-token\") pod \"009b85b2-8630-42b6-875a-ed084b0593b6\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " Sep 08 00:50:24.721894 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.721784 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009b85b2-8630-42b6-875a-ed084b0593b6-istiod-ca-cert\") pod \"009b85b2-8630-42b6-875a-ed084b0593b6\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " Sep 08 00:50:24.722152 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.721890 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009b85b2-8630-42b6-875a-ed084b0593b6-istio-podinfo\") pod \"009b85b2-8630-42b6-875a-ed084b0593b6\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " Sep 08 00:50:24.722152 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.721908 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-istio-data" (OuterVolumeSpecName: "istio-data") pod "009b85b2-8630-42b6-875a-ed084b0593b6" (UID: "009b85b2-8630-42b6-875a-ed084b0593b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:50:24.722152 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.721941 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-workload-certs\") pod \"009b85b2-8630-42b6-875a-ed084b0593b6\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " Sep 08 00:50:24.722152 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.721957 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "009b85b2-8630-42b6-875a-ed084b0593b6" (UID: "009b85b2-8630-42b6-875a-ed084b0593b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:50:24.722152 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.722037 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-workload-socket\") pod \"009b85b2-8630-42b6-875a-ed084b0593b6\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " Sep 08 00:50:24.722152 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.722091 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w6nh\" (UniqueName: \"kubernetes.io/projected/009b85b2-8630-42b6-875a-ed084b0593b6-kube-api-access-4w6nh\") pod \"009b85b2-8630-42b6-875a-ed084b0593b6\" (UID: \"009b85b2-8630-42b6-875a-ed084b0593b6\") " Sep 08 00:50:24.722152 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.722130 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/009b85b2-8630-42b6-875a-ed084b0593b6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "009b85b2-8630-42b6-875a-ed084b0593b6" (UID: "009b85b2-8630-42b6-875a-ed084b0593b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:50:24.722494 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.722173 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "009b85b2-8630-42b6-875a-ed084b0593b6" (UID: "009b85b2-8630-42b6-875a-ed084b0593b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:50:24.722494 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.722190 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "009b85b2-8630-42b6-875a-ed084b0593b6" (UID: "009b85b2-8630-42b6-875a-ed084b0593b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:50:24.722494 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.722409 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-istio-data\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:50:24.722494 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.722429 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-credential-socket\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:50:24.722494 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.722442 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009b85b2-8630-42b6-875a-ed084b0593b6-istiod-ca-cert\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:50:24.722494 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.722455 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-workload-certs\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:50:24.722494 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.722468 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-workload-socket\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:50:24.724169 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.724137 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "009b85b2-8630-42b6-875a-ed084b0593b6" (UID: "009b85b2-8630-42b6-875a-ed084b0593b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:50:24.724169 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.724150 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009b85b2-8630-42b6-875a-ed084b0593b6-istio-token" (OuterVolumeSpecName: "istio-token") pod "009b85b2-8630-42b6-875a-ed084b0593b6" (UID: "009b85b2-8630-42b6-875a-ed084b0593b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:50:24.724313 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.724188 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009b85b2-8630-42b6-875a-ed084b0593b6-kube-api-access-4w6nh" (OuterVolumeSpecName: "kube-api-access-4w6nh") pod "009b85b2-8630-42b6-875a-ed084b0593b6" (UID: "009b85b2-8630-42b6-875a-ed084b0593b6"). InnerVolumeSpecName "kube-api-access-4w6nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:50:24.724313 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.724195 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/009b85b2-8630-42b6-875a-ed084b0593b6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "009b85b2-8630-42b6-875a-ed084b0593b6" (UID: "009b85b2-8630-42b6-875a-ed084b0593b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 00:50:24.820581 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.820547 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:50:24.820785 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.820762 2577 patch_prober.go:28] interesting pod/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 08 00:50:24.820832 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.820812 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" podUID="0bd1765e-c6fd-4b46-8748-ccaf159d7f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 08 00:50:24.823500 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.823473 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009b85b2-8630-42b6-875a-ed084b0593b6-istio-envoy\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:50:24.823500 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.823502 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009b85b2-8630-42b6-875a-ed084b0593b6-istio-token\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:50:24.823716 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.823518 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009b85b2-8630-42b6-875a-ed084b0593b6-istio-podinfo\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:50:24.823716 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.823531 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w6nh\" (UniqueName: \"kubernetes.io/projected/009b85b2-8630-42b6-875a-ed084b0593b6-kube-api-access-4w6nh\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:50:24.980051 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.980018 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw"] Sep 08 00:50:24.983126 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:24.983097 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e3896b-openshift-default-5b6cbf65b6-g4dzw"] Sep 08 00:50:25.349657 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.349617 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9"] Sep 08 00:50:25.350047 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.350029 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="009b85b2-8630-42b6-875a-ed084b0593b6" containerName="istio-proxy" Sep 08 00:50:25.350146 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.350048 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="009b85b2-8630-42b6-875a-ed084b0593b6" containerName="istio-proxy" Sep 08 00:50:25.350146 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.350142 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="009b85b2-8630-42b6-875a-ed084b0593b6" containerName="istio-proxy" Sep 08 00:50:25.350259 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.350155 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="009b85b2-8630-42b6-875a-ed084b0593b6" containerName="istio-proxy" Sep 08 00:50:25.353083 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.353059 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9" Sep 08 00:50:25.356672 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.356650 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-b8c4de\"" Sep 08 00:50:25.356868 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.356846 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-b8c4de-dockercfg-d2hhv\"" Sep 08 00:50:25.363620 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.363594 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9"] Sep 08 00:50:25.377444 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.377412 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq"] Sep 08 00:50:25.377915 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.377898 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="009b85b2-8630-42b6-875a-ed084b0593b6" containerName="istio-proxy" Sep 08 00:50:25.377966 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.377921 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="009b85b2-8630-42b6-875a-ed084b0593b6" containerName="istio-proxy" Sep 08 00:50:25.381077 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.381060 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq" Sep 08 00:50:25.395025 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.391721 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq"] Sep 08 00:50:25.429463 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.429426 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ed17de-8d86-403c-8d99-992f1c82a501-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq\" (UID: \"45ed17de-8d86-403c-8d99-992f1c82a501\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq" Sep 08 00:50:25.429637 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.429470 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqhd7\" (UniqueName: \"kubernetes.io/projected/45ed17de-8d86-403c-8d99-992f1c82a501-kube-api-access-dqhd7\") pod \"payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq\" (UID: \"45ed17de-8d86-403c-8d99-992f1c82a501\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq" Sep 08 00:50:25.429637 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.429519 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2cpm\" (UniqueName: \"kubernetes.io/projected/962c714d-8126-4564-99bc-319dd04b4c10-kube-api-access-f2cpm\") pod \"payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9\" (UID: \"962c714d-8126-4564-99bc-319dd04b4c10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9" Sep 08 00:50:25.429637 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.429551 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/962c714d-8126-4564-99bc-319dd04b4c10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9\" (UID: \"962c714d-8126-4564-99bc-319dd04b4c10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9" Sep 08 00:50:25.530661 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.530622 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/962c714d-8126-4564-99bc-319dd04b4c10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9\" (UID: \"962c714d-8126-4564-99bc-319dd04b4c10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9" Sep 08 00:50:25.530872 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.530702 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ed17de-8d86-403c-8d99-992f1c82a501-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq\" (UID: \"45ed17de-8d86-403c-8d99-992f1c82a501\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq" Sep 08 00:50:25.530872 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.530734 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqhd7\" (UniqueName: \"kubernetes.io/projected/45ed17de-8d86-403c-8d99-992f1c82a501-kube-api-access-dqhd7\") pod \"payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq\" (UID: \"45ed17de-8d86-403c-8d99-992f1c82a501\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq" Sep 08 00:50:25.530872 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.530768 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2cpm\" (UniqueName: \"kubernetes.io/projected/962c714d-8126-4564-99bc-319dd04b4c10-kube-api-access-f2cpm\") pod \"payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9\" (UID: \"962c714d-8126-4564-99bc-319dd04b4c10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9" Sep 08 00:50:25.531474 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.531449 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/962c714d-8126-4564-99bc-319dd04b4c10-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9\" (UID: \"962c714d-8126-4564-99bc-319dd04b4c10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9" Sep 08 00:50:25.531474 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.531463 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ed17de-8d86-403c-8d99-992f1c82a501-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq\" (UID: \"45ed17de-8d86-403c-8d99-992f1c82a501\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq" Sep 08 00:50:25.538523 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.538504 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2cpm\" (UniqueName: \"kubernetes.io/projected/962c714d-8126-4564-99bc-319dd04b4c10-kube-api-access-f2cpm\") pod \"payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9\" (UID: \"962c714d-8126-4564-99bc-319dd04b4c10\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9" Sep 08 00:50:25.538620 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.538508 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqhd7\" (UniqueName: \"kubernetes.io/projected/45ed17de-8d86-403c-8d99-992f1c82a501-kube-api-access-dqhd7\") pod \"payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq\" (UID: \"45ed17de-8d86-403c-8d99-992f1c82a501\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq" Sep 08 00:50:25.662903 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.662809 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9" Sep 08 00:50:25.713280 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.713190 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq" Sep 08 00:50:25.821060 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.820959 2577 patch_prober.go:28] interesting pod/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 08 00:50:25.821060 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.821048 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" podUID="0bd1765e-c6fd-4b46-8748-ccaf159d7f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 08 00:50:25.958291 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.958266 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9"] Sep 08 00:50:25.960298 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:50:25.960266 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod962c714d_8126_4564_99bc_319dd04b4c10.slice/crio-87dc947dfe7489c0af2a9e47d3efcc6cc75406769616dee553e658587fde5abf WatchSource:0}: Error finding container 87dc947dfe7489c0af2a9e47d3efcc6cc75406769616dee553e658587fde5abf: Status 404 returned error can't find the container with id 87dc947dfe7489c0af2a9e47d3efcc6cc75406769616dee553e658587fde5abf Sep 08 00:50:26.000016 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:25.999081 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq"] Sep 08 00:50:26.009573 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:50:26.009534 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ed17de_8d86_403c_8d99_992f1c82a501.slice/crio-707ceb227ce1ee6104e14a4aa44f69cccf76def6291f1fc2956879676ea8f721 WatchSource:0}: Error finding container 707ceb227ce1ee6104e14a4aa44f69cccf76def6291f1fc2956879676ea8f721: Status 404 returned error can't find the container with id 707ceb227ce1ee6104e14a4aa44f69cccf76def6291f1fc2956879676ea8f721 Sep 08 00:50:26.675375 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:26.675334 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9" event={"ID":"962c714d-8126-4564-99bc-319dd04b4c10","Type":"ContainerStarted","Data":"36ec4ae815b2521b83abb9c642fb1858e47cfe2ec7155c7076240c494a7f428a"} Sep 08 00:50:26.675855 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:26.675405 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9" Sep 08 00:50:26.675855 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:26.675432 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9" event={"ID":"962c714d-8126-4564-99bc-319dd04b4c10","Type":"ContainerStarted","Data":"87dc947dfe7489c0af2a9e47d3efcc6cc75406769616dee553e658587fde5abf"} Sep 08 00:50:26.677024 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:26.676981 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq" event={"ID":"45ed17de-8d86-403c-8d99-992f1c82a501","Type":"ContainerStarted","Data":"b16458a423c14fa05cc0e77e9e8cf38c7aec65f76003ec8d6bd99737ec19cacd"} Sep 08 00:50:26.677145 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:26.677049 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq" event={"ID":"45ed17de-8d86-403c-8d99-992f1c82a501","Type":"ContainerStarted","Data":"707ceb227ce1ee6104e14a4aa44f69cccf76def6291f1fc2956879676ea8f721"} Sep 08 00:50:26.677145 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:26.677078 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq" Sep 08 00:50:26.693837 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:26.693763 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9" podStartSLOduration=1.693745026 podStartE2EDuration="1.693745026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:50:26.693397081 +0000 UTC m=+1040.388638241" watchObservedRunningTime="2026-09-08 00:50:26.693745026 +0000 UTC m=+1040.388986187" Sep 08 00:50:26.792057 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:26.792020 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="009b85b2-8630-42b6-875a-ed084b0593b6" path="/var/lib/kubelet/pods/009b85b2-8630-42b6-875a-ed084b0593b6/volumes" Sep 08 00:50:26.821714 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:26.821338 2577 patch_prober.go:28] interesting pod/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 08 00:50:26.821714 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:26.821424 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" podUID="0bd1765e-c6fd-4b46-8748-ccaf159d7f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 08 00:50:27.820710 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:27.820665 2577 patch_prober.go:28] interesting pod/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 08 00:50:27.821155 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:27.820729 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" podUID="0bd1765e-c6fd-4b46-8748-ccaf159d7f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 08 00:50:28.820330 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:28.820295 2577 patch_prober.go:28] interesting pod/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Sep 08 00:50:28.820515 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:28.820359 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" podUID="0bd1765e-c6fd-4b46-8748-ccaf159d7f84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Sep 08 00:50:29.860313 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:29.860280 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:50:29.860977 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:29.860949 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:50:29.862090 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:29.862070 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:50:29.883494 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:29.883442 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq" podStartSLOduration=4.883428898 podStartE2EDuration="4.883428898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:50:26.715655179 +0000 UTC m=+1040.410896339" watchObservedRunningTime="2026-09-08 00:50:29.883428898 +0000 UTC m=+1043.578670056" Sep 08 00:50:37.683555 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:37.683464 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9" Sep 08 00:50:37.683953 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:37.683783 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq" Sep 08 00:50:38.688188 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:38.688145 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 08 00:50:38.688613 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:38.688215 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" podUID="cb68d759-3712-4438-8ed8-19b3e331c7b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 08 00:50:40.756507 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:40.756414 2577 generic.go:356] "Generic (PLEG): container finished" podID="cb68d759-3712-4438-8ed8-19b3e331c7b5" containerID="24720273b54c3d0008ee8ecfdbbf7fadd15bc7077ff2a26b2328c34624b6aefc" exitCode=0 Sep 08 00:50:40.756507 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:40.756491 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" event={"ID":"cb68d759-3712-4438-8ed8-19b3e331c7b5","Type":"ContainerDied","Data":"24720273b54c3d0008ee8ecfdbbf7fadd15bc7077ff2a26b2328c34624b6aefc"} Sep 08 00:50:40.756952 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:40.756933 2577 scope.go:117] "RemoveContainer" containerID="24720273b54c3d0008ee8ecfdbbf7fadd15bc7077ff2a26b2328c34624b6aefc" Sep 08 00:50:40.758047 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:40.757990 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:50:40.758103 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:40.758089 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:50:40.758142 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:40.758120 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 08 00:50:41.763369 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:41.763333 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" event={"ID":"cb68d759-3712-4438-8ed8-19b3e331c7b5","Type":"ContainerStarted","Data":"592bf287fc6227350cdbf69d77b5b423806a95d28fd507b5c61fbdba6e60a927"} Sep 08 00:50:42.354197 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:42.354160 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:42.354403 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:42.354378 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 08 00:50:42.354471 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:42.354445 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" podUID="cb68d759-3712-4438-8ed8-19b3e331c7b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 08 00:50:43.354955 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:43.354921 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 08 00:50:43.355362 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:43.354985 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" podUID="cb68d759-3712-4438-8ed8-19b3e331c7b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 08 00:50:44.354257 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:44.354218 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 08 00:50:44.354464 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:44.354281 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" podUID="cb68d759-3712-4438-8ed8-19b3e331c7b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 08 00:50:45.354614 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:45.354580 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 08 00:50:45.355011 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:45.354637 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" podUID="cb68d759-3712-4438-8ed8-19b3e331c7b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 08 00:50:46.354867 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:46.354777 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Sep 08 00:50:46.354867 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:46.354833 2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" podUID="cb68d759-3712-4438-8ed8-19b3e331c7b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Sep 08 00:50:47.358466 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:47.358434 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:47.358838 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:47.358716 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:47.359455 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:47.359436 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:50:53.889527 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:53.889490 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6"] Sep 08 00:50:53.892471 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:53.892452 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6" Sep 08 00:50:53.895636 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:53.895615 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-a8e810-dockercfg-fbrqw\"" Sep 08 00:50:53.895776 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:53.895654 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-a8e810\"" Sep 08 00:50:53.905817 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:53.905793 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6"] Sep 08 00:50:53.999898 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:53.999863 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v79vg\" (UniqueName: \"kubernetes.io/projected/a43afd79-ff45-4dd6-b9f9-9fb355565a0c-kube-api-access-v79vg\") pod \"payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6\" (UID: \"a43afd79-ff45-4dd6-b9f9-9fb355565a0c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6" Sep 08 00:50:54.000109 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:54.000030 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43afd79-ff45-4dd6-b9f9-9fb355565a0c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6\" (UID: \"a43afd79-ff45-4dd6-b9f9-9fb355565a0c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6" Sep 08 00:50:54.101090 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:54.101052 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43afd79-ff45-4dd6-b9f9-9fb355565a0c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6\" (UID: \"a43afd79-ff45-4dd6-b9f9-9fb355565a0c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6" Sep 08 00:50:54.101288 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:54.101141 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v79vg\" (UniqueName: \"kubernetes.io/projected/a43afd79-ff45-4dd6-b9f9-9fb355565a0c-kube-api-access-v79vg\") pod \"payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6\" (UID: \"a43afd79-ff45-4dd6-b9f9-9fb355565a0c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6" Sep 08 00:50:54.101825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:54.101797 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43afd79-ff45-4dd6-b9f9-9fb355565a0c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6\" (UID: \"a43afd79-ff45-4dd6-b9f9-9fb355565a0c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6" Sep 08 00:50:54.109679 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:54.109649 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v79vg\" (UniqueName: \"kubernetes.io/projected/a43afd79-ff45-4dd6-b9f9-9fb355565a0c-kube-api-access-v79vg\") pod \"payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6\" (UID: \"a43afd79-ff45-4dd6-b9f9-9fb355565a0c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6" Sep 08 00:50:54.202221 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:54.202133 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6" Sep 08 00:50:54.470523 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:54.470498 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6"] Sep 08 00:50:54.472603 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:50:54.472565 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43afd79_ff45_4dd6_b9f9_9fb355565a0c.slice/crio-628dfdb79448853ec8e0ff2a31bfa0c97f9c80251088e9731b1ef1d365ec969c WatchSource:0}: Error finding container 628dfdb79448853ec8e0ff2a31bfa0c97f9c80251088e9731b1ef1d365ec969c: Status 404 returned error can't find the container with id 628dfdb79448853ec8e0ff2a31bfa0c97f9c80251088e9731b1ef1d365ec969c Sep 08 00:50:54.828912 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:54.828877 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6" event={"ID":"a43afd79-ff45-4dd6-b9f9-9fb355565a0c","Type":"ContainerStarted","Data":"d8ccfde54b8b125e8396a389029533780cd1c4c973f02fc177f038e040f8e6d4"} Sep 08 00:50:54.829138 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:54.828932 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6" event={"ID":"a43afd79-ff45-4dd6-b9f9-9fb355565a0c","Type":"ContainerStarted","Data":"628dfdb79448853ec8e0ff2a31bfa0c97f9c80251088e9731b1ef1d365ec969c"} Sep 08 00:50:54.829138 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:54.829022 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6" Sep 08 00:50:54.845167 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:50:54.845118 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6" podStartSLOduration=1.8451031439999999 podStartE2EDuration="1.845103144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:50:54.843532786 +0000 UTC m=+1068.538773946" watchObservedRunningTime="2026-09-08 00:50:54.845103144 +0000 UTC m=+1068.540344342" Sep 08 00:51:05.837190 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:05.837158 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6" Sep 08 00:51:06.833553 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:06.833521 2577 scope.go:117] "RemoveContainer" containerID="8f5ba08711b6ebb476b9ef34873c5925f37584d94dfc1ce1bd0fef6053bb057a" Sep 08 00:51:06.841790 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:06.841770 2577 scope.go:117] "RemoveContainer" containerID="1db335abf7eeae7936921517bf1d7aba724b9f5036e8bccffbdf81927d2e3af5" Sep 08 00:51:06.849455 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:06.849430 2577 scope.go:117] "RemoveContainer" containerID="5fa8e448d96467042edb55156c527bab5614deaf0e765a3c1e1efb35e6318c12" Sep 08 00:51:41.109589 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:41.109549 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c28b2155-lgxr9"] Sep 08 00:51:41.112779 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:41.112759 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c28b2155-lgxr9" Sep 08 00:51:41.115475 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:41.115453 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 00:51:41.115603 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:41.115460 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dqw2\"" Sep 08 00:51:41.119435 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:41.119410 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c28b2155-lgxr9"] Sep 08 00:51:41.217157 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:41.217120 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b422492-d061-4aac-ab5b-0ca5e45ba0eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c28b2155-lgxr9\" (UID: \"9b422492-d061-4aac-ab5b-0ca5e45ba0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c28b2155-lgxr9" Sep 08 00:51:41.318393 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:41.318355 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b422492-d061-4aac-ab5b-0ca5e45ba0eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c28b2155-lgxr9\" (UID: \"9b422492-d061-4aac-ab5b-0ca5e45ba0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c28b2155-lgxr9" Sep 08 00:51:41.318967 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:41.318947 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b422492-d061-4aac-ab5b-0ca5e45ba0eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c28b2155-lgxr9\" (UID: \"9b422492-d061-4aac-ab5b-0ca5e45ba0eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c28b2155-lgxr9" Sep 08 00:51:41.424704 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:41.424620 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c28b2155-lgxr9" Sep 08 00:51:41.692203 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:41.692119 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c28b2155-lgxr9"] Sep 08 00:51:41.694480 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:51:41.694451 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b422492_d061_4aac_ab5b_0ca5e45ba0eb.slice/crio-5c87b145b9ae6dde34c60b04ca686e179ac8f1d6feaa1099fac15e210596cff9 WatchSource:0}: Error finding container 5c87b145b9ae6dde34c60b04ca686e179ac8f1d6feaa1099fac15e210596cff9: Status 404 returned error can't find the container with id 5c87b145b9ae6dde34c60b04ca686e179ac8f1d6feaa1099fac15e210596cff9 Sep 08 00:51:42.073369 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:42.073333 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c28b2155-lgxr9" event={"ID":"9b422492-d061-4aac-ab5b-0ca5e45ba0eb","Type":"ContainerStarted","Data":"ac1a9ad99838e5d37dd08886847e0268abe43695d319b719e6559b0e517726c6"} Sep 08 00:51:42.073579 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:42.073408 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c28b2155-lgxr9" event={"ID":"9b422492-d061-4aac-ab5b-0ca5e45ba0eb","Type":"ContainerStarted","Data":"5c87b145b9ae6dde34c60b04ca686e179ac8f1d6feaa1099fac15e210596cff9"} Sep 08 00:51:42.088222 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:42.088173 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c28b2155-lgxr9" podStartSLOduration=1.088159445 podStartE2EDuration="1.088159445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:51:42.086508718 +0000 UTC m=+1115.781749880" watchObservedRunningTime="2026-09-08 00:51:42.088159445 +0000 UTC m=+1115.783400603" Sep 08 00:51:43.079689 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:43.079652 2577 generic.go:356] "Generic (PLEG): container finished" podID="9b422492-d061-4aac-ab5b-0ca5e45ba0eb" containerID="ac1a9ad99838e5d37dd08886847e0268abe43695d319b719e6559b0e517726c6" exitCode=0 Sep 08 00:51:43.080089 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:43.079694 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c28b2155-lgxr9" event={"ID":"9b422492-d061-4aac-ab5b-0ca5e45ba0eb","Type":"ContainerDied","Data":"ac1a9ad99838e5d37dd08886847e0268abe43695d319b719e6559b0e517726c6"} Sep 08 00:51:44.211920 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:44.211895 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c28b2155-lgxr9" Sep 08 00:51:44.345188 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:44.345075 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b422492-d061-4aac-ab5b-0ca5e45ba0eb-maas-api-service-ca\") pod \"9b422492-d061-4aac-ab5b-0ca5e45ba0eb\" (UID: \"9b422492-d061-4aac-ab5b-0ca5e45ba0eb\") " Sep 08 00:51:44.345485 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:44.345455 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b422492-d061-4aac-ab5b-0ca5e45ba0eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9b422492-d061-4aac-ab5b-0ca5e45ba0eb" (UID: "9b422492-d061-4aac-ab5b-0ca5e45ba0eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:51:44.445850 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:44.445815 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b422492-d061-4aac-ab5b-0ca5e45ba0eb-maas-api-service-ca\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:51:45.091959 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:45.091864 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c28b2155-lgxr9" Sep 08 00:51:45.091959 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:45.091881 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c28b2155-lgxr9" event={"ID":"9b422492-d061-4aac-ab5b-0ca5e45ba0eb","Type":"ContainerDied","Data":"5c87b145b9ae6dde34c60b04ca686e179ac8f1d6feaa1099fac15e210596cff9"} Sep 08 00:51:45.091959 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:45.091941 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c87b145b9ae6dde34c60b04ca686e179ac8f1d6feaa1099fac15e210596cff9" Sep 08 00:51:56.800759 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:56.800725 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c28b2155-lgxr9"] Sep 08 00:51:56.802384 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:56.802363 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c28b2155-lgxr9"] Sep 08 00:51:58.790078 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:51:58.790034 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b422492-d061-4aac-ab5b-0ca5e45ba0eb" path="/var/lib/kubelet/pods/9b422492-d061-4aac-ab5b-0ca5e45ba0eb/volumes" Sep 08 00:52:06.891789 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:06.891759 2577 scope.go:117] "RemoveContainer" containerID="2c1a416bba4c63340074c29ffb33047e11ae95be56a9154aba00c12c058fea12" Sep 08 00:52:06.901844 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:06.901820 2577 scope.go:117] "RemoveContainer" containerID="06648f903a1d3e5cd9bf5d87f84c7dddc324b430bf18276b27fa49ee0be77779" Sep 08 00:52:06.909277 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:06.909256 2577 scope.go:117] "RemoveContainer" containerID="057975b39394563756a1f7943e493bc2b860f7da0c0c5369763d0640d391977e" Sep 08 00:52:47.546099 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:47.546062 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5"] Sep 08 00:52:47.546548 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:47.546272 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" podUID="0bd1765e-c6fd-4b46-8748-ccaf159d7f84" containerName="istio-proxy" containerID="cri-o://c7dd66addaa88af7786778d2ac76c1f7dcdca45ad0d7a7e74810a26148838557" gracePeriod=30 Sep 08 00:52:52.789683 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.789654 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:52:52.841143 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.841064 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-credential-socket\") pod \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " Sep 08 00:52:52.841143 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.841099 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqhmc\" (UniqueName: \"kubernetes.io/projected/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-kube-api-access-wqhmc\") pod \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " Sep 08 00:52:52.841367 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.841158 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-workload-certs\") pod \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " Sep 08 00:52:52.841367 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.841190 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-token\") pod \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " Sep 08 00:52:52.841367 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.841225 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istiod-ca-cert\") pod \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " Sep 08 00:52:52.841367 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.841262 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-data\") pod \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " Sep 08 00:52:52.841367 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.841291 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-podinfo\") pod \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " Sep 08 00:52:52.841367 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.841340 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-envoy\") pod \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " Sep 08 00:52:52.841367 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.841365 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-workload-socket\") pod \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\" (UID: \"0bd1765e-c6fd-4b46-8748-ccaf159d7f84\") " Sep 08 00:52:52.841737 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.841385 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0bd1765e-c6fd-4b46-8748-ccaf159d7f84" (UID: "0bd1765e-c6fd-4b46-8748-ccaf159d7f84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:52:52.841737 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.841480 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0bd1765e-c6fd-4b46-8748-ccaf159d7f84" (UID: "0bd1765e-c6fd-4b46-8748-ccaf159d7f84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:52:52.841737 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.841672 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-credential-socket\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:52:52.841737 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.841692 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-workload-certs\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:52:52.841944 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.841699 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0bd1765e-c6fd-4b46-8748-ccaf159d7f84" (UID: "0bd1765e-c6fd-4b46-8748-ccaf159d7f84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:52:52.841944 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.841737 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0bd1765e-c6fd-4b46-8748-ccaf159d7f84" (UID: "0bd1765e-c6fd-4b46-8748-ccaf159d7f84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:52:52.841944 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.841891 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-data" (OuterVolumeSpecName: "istio-data") pod "0bd1765e-c6fd-4b46-8748-ccaf159d7f84" (UID: "0bd1765e-c6fd-4b46-8748-ccaf159d7f84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:52:52.843520 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.843495 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-token" (OuterVolumeSpecName: "istio-token") pod "0bd1765e-c6fd-4b46-8748-ccaf159d7f84" (UID: "0bd1765e-c6fd-4b46-8748-ccaf159d7f84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:52:52.843520 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.843515 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0bd1765e-c6fd-4b46-8748-ccaf159d7f84" (UID: "0bd1765e-c6fd-4b46-8748-ccaf159d7f84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:52:52.843732 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.843716 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0bd1765e-c6fd-4b46-8748-ccaf159d7f84" (UID: "0bd1765e-c6fd-4b46-8748-ccaf159d7f84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 00:52:52.843793 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.843770 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-kube-api-access-wqhmc" (OuterVolumeSpecName: "kube-api-access-wqhmc") pod "0bd1765e-c6fd-4b46-8748-ccaf159d7f84" (UID: "0bd1765e-c6fd-4b46-8748-ccaf159d7f84"). InnerVolumeSpecName "kube-api-access-wqhmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:52:52.942801 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.942768 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-token\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:52:52.942801 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.942796 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istiod-ca-cert\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:52:52.942801 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.942805 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-data\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:52:52.943048 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.942813 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-podinfo\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:52:52.943048 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.942821 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-istio-envoy\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:52:52.943048 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.942829 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-workload-socket\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:52:52.943048 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:52.942838 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqhmc\" (UniqueName: \"kubernetes.io/projected/0bd1765e-c6fd-4b46-8748-ccaf159d7f84-kube-api-access-wqhmc\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:52:53.425083 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:53.425047 2577 generic.go:356] "Generic (PLEG): container finished" podID="0bd1765e-c6fd-4b46-8748-ccaf159d7f84" containerID="c7dd66addaa88af7786778d2ac76c1f7dcdca45ad0d7a7e74810a26148838557" exitCode=0 Sep 08 00:52:53.425293 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:53.425104 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" Sep 08 00:52:53.425293 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:53.425124 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" event={"ID":"0bd1765e-c6fd-4b46-8748-ccaf159d7f84","Type":"ContainerDied","Data":"c7dd66addaa88af7786778d2ac76c1f7dcdca45ad0d7a7e74810a26148838557"} Sep 08 00:52:53.425293 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:53.425165 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5" event={"ID":"0bd1765e-c6fd-4b46-8748-ccaf159d7f84","Type":"ContainerDied","Data":"04304beba3da41cd4366e6be0a5ff800a7e790ec8fcb5c1c61da56b10cf583fb"} Sep 08 00:52:53.425293 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:53.425186 2577 scope.go:117] "RemoveContainer" containerID="c7dd66addaa88af7786778d2ac76c1f7dcdca45ad0d7a7e74810a26148838557" Sep 08 00:52:53.434558 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:53.434540 2577 scope.go:117] "RemoveContainer" containerID="e6dda66b30c94f7157738b93433cbd0167e6b9de78ef23ce4dfd24e856e1df32" Sep 08 00:52:53.442019 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:53.441986 2577 scope.go:117] "RemoveContainer" containerID="c7dd66addaa88af7786778d2ac76c1f7dcdca45ad0d7a7e74810a26148838557" Sep 08 00:52:53.442291 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:52:53.442268 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7dd66addaa88af7786778d2ac76c1f7dcdca45ad0d7a7e74810a26148838557\": container with ID starting with c7dd66addaa88af7786778d2ac76c1f7dcdca45ad0d7a7e74810a26148838557 not found: ID does not exist" containerID="c7dd66addaa88af7786778d2ac76c1f7dcdca45ad0d7a7e74810a26148838557" Sep 08 00:52:53.442435 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:53.442303 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7dd66addaa88af7786778d2ac76c1f7dcdca45ad0d7a7e74810a26148838557"} err="failed to get container status \"c7dd66addaa88af7786778d2ac76c1f7dcdca45ad0d7a7e74810a26148838557\": rpc error: code = NotFound desc = could not find container \"c7dd66addaa88af7786778d2ac76c1f7dcdca45ad0d7a7e74810a26148838557\": container with ID starting with c7dd66addaa88af7786778d2ac76c1f7dcdca45ad0d7a7e74810a26148838557 not found: ID does not exist" Sep 08 00:52:53.442435 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:53.442326 2577 scope.go:117] "RemoveContainer" containerID="e6dda66b30c94f7157738b93433cbd0167e6b9de78ef23ce4dfd24e856e1df32" Sep 08 00:52:53.442602 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:52:53.442583 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6dda66b30c94f7157738b93433cbd0167e6b9de78ef23ce4dfd24e856e1df32\": container with ID starting with e6dda66b30c94f7157738b93433cbd0167e6b9de78ef23ce4dfd24e856e1df32 not found: ID does not exist" containerID="e6dda66b30c94f7157738b93433cbd0167e6b9de78ef23ce4dfd24e856e1df32" Sep 08 00:52:53.442642 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:53.442611 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6dda66b30c94f7157738b93433cbd0167e6b9de78ef23ce4dfd24e856e1df32"} err="failed to get container status \"e6dda66b30c94f7157738b93433cbd0167e6b9de78ef23ce4dfd24e856e1df32\": rpc error: code = NotFound desc = could not find container \"e6dda66b30c94f7157738b93433cbd0167e6b9de78ef23ce4dfd24e856e1df32\": container with ID starting with e6dda66b30c94f7157738b93433cbd0167e6b9de78ef23ce4dfd24e856e1df32 not found: ID does not exist" Sep 08 00:52:53.445804 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:53.445778 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5"] Sep 08 00:52:53.448157 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:53.448136 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c28b2155-openshift-default-679d9844d-7jlq5"] Sep 08 00:52:54.790004 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:52:54.789958 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bd1765e-c6fd-4b46-8748-ccaf159d7f84" path="/var/lib/kubelet/pods/0bd1765e-c6fd-4b46-8748-ccaf159d7f84/volumes" Sep 08 00:53:06.753565 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:06.753537 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9nb9x_a2e160ae-d75d-4959-a5ab-d238b4113736/console-operator/1.log" Sep 08 00:53:06.755963 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:06.755942 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9nb9x_a2e160ae-d75d-4959-a5ab-d238b4113736/console-operator/1.log" Sep 08 00:53:46.062507 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:46.062473 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm"] Sep 08 00:53:46.063124 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:46.062968 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bd1765e-c6fd-4b46-8748-ccaf159d7f84" containerName="istio-proxy" Sep 08 00:53:46.063124 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:46.062987 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bd1765e-c6fd-4b46-8748-ccaf159d7f84" containerName="istio-proxy" Sep 08 00:53:46.063124 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:46.063029 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b422492-d061-4aac-ab5b-0ca5e45ba0eb" containerName="revoke-keys" Sep 08 00:53:46.063124 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:46.063039 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b422492-d061-4aac-ab5b-0ca5e45ba0eb" containerName="revoke-keys" Sep 08 00:53:46.063124 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:46.063124 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bd1765e-c6fd-4b46-8748-ccaf159d7f84" containerName="istio-proxy" Sep 08 00:53:46.063416 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:46.063137 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bd1765e-c6fd-4b46-8748-ccaf159d7f84" containerName="istio-proxy" Sep 08 00:53:46.063416 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:46.063148 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b422492-d061-4aac-ab5b-0ca5e45ba0eb" containerName="revoke-keys" Sep 08 00:53:46.066022 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:46.065977 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm" Sep 08 00:53:46.068905 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:46.068882 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-55b1e9-dockercfg-frd7n\"" Sep 08 00:53:46.069036 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:46.068928 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-55b1e9\"" Sep 08 00:53:46.076436 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:46.076415 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm"] Sep 08 00:53:46.104140 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:46.104112 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5148709d-5baf-4a04-bf5a-a0a8b36edced-plugins-config-volume\") pod \"payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm\" (UID: \"5148709d-5baf-4a04-bf5a-a0a8b36edced\") " pod="openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm" Sep 08 00:53:46.104252 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:46.104197 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ftft\" (UniqueName: \"kubernetes.io/projected/5148709d-5baf-4a04-bf5a-a0a8b36edced-kube-api-access-4ftft\") pod \"payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm\" (UID: \"5148709d-5baf-4a04-bf5a-a0a8b36edced\") " pod="openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm" Sep 08 00:53:46.204899 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:46.204863 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ftft\" (UniqueName: \"kubernetes.io/projected/5148709d-5baf-4a04-bf5a-a0a8b36edced-kube-api-access-4ftft\") pod \"payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm\" (UID: \"5148709d-5baf-4a04-bf5a-a0a8b36edced\") " pod="openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm" Sep 08 00:53:46.205120 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:46.204908 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5148709d-5baf-4a04-bf5a-a0a8b36edced-plugins-config-volume\") pod \"payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm\" (UID: \"5148709d-5baf-4a04-bf5a-a0a8b36edced\") " pod="openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm" Sep 08 00:53:46.205663 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:46.205638 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5148709d-5baf-4a04-bf5a-a0a8b36edced-plugins-config-volume\") pod \"payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm\" (UID: \"5148709d-5baf-4a04-bf5a-a0a8b36edced\") " pod="openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm" Sep 08 00:53:46.212636 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:46.212614 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ftft\" (UniqueName: \"kubernetes.io/projected/5148709d-5baf-4a04-bf5a-a0a8b36edced-kube-api-access-4ftft\") pod \"payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm\" (UID: \"5148709d-5baf-4a04-bf5a-a0a8b36edced\") " pod="openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm" Sep 08 00:53:46.377643 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:46.377546 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm" Sep 08 00:53:46.644942 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:46.644916 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm"] Sep 08 00:53:46.646973 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:53:46.646941 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5148709d_5baf_4a04_bf5a_a0a8b36edced.slice/crio-30f4071b5c6f3914f3274c90a05c4e7e9872a89e56e3f76b759326ef1ac868d1 WatchSource:0}: Error finding container 30f4071b5c6f3914f3274c90a05c4e7e9872a89e56e3f76b759326ef1ac868d1: Status 404 returned error can't find the container with id 30f4071b5c6f3914f3274c90a05c4e7e9872a89e56e3f76b759326ef1ac868d1 Sep 08 00:53:46.675798 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:46.675774 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm" event={"ID":"5148709d-5baf-4a04-bf5a-a0a8b36edced","Type":"ContainerStarted","Data":"30f4071b5c6f3914f3274c90a05c4e7e9872a89e56e3f76b759326ef1ac868d1"} Sep 08 00:53:47.682839 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:47.682804 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm" event={"ID":"5148709d-5baf-4a04-bf5a-a0a8b36edced","Type":"ContainerStarted","Data":"312f86fbeab012a8f4abce2b6d0379bb4c4ab3947827793c28fddde3d3c26d3e"} Sep 08 00:53:47.683232 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:47.682881 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm" Sep 08 00:53:47.700588 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:47.700533 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm" podStartSLOduration=1.700518148 podStartE2EDuration="1.700518148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:53:47.699236397 +0000 UTC m=+1241.394477557" watchObservedRunningTime="2026-09-08 00:53:47.700518148 +0000 UTC m=+1241.395759307" Sep 08 00:53:58.690573 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:53:58.690542 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm" Sep 08 00:54:15.150640 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.150605 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk"] Sep 08 00:54:15.151102 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.150922 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bd1765e-c6fd-4b46-8748-ccaf159d7f84" containerName="istio-proxy" Sep 08 00:54:15.151102 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.150933 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bd1765e-c6fd-4b46-8748-ccaf159d7f84" containerName="istio-proxy" Sep 08 00:54:15.153961 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.153944 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk" Sep 08 00:54:15.156515 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.156494 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3e8800-dockercfg-mpx8s\"" Sep 08 00:54:15.156695 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.156678 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3e8800\"" Sep 08 00:54:15.164695 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.164671 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk"] Sep 08 00:54:15.193481 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.193453 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl"] Sep 08 00:54:15.196688 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.196673 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl" Sep 08 00:54:15.207339 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.207317 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl"] Sep 08 00:54:15.250204 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.250159 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9375e6a3-401f-48d6-bb77-c37a0585ea95-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl\" (UID: \"9375e6a3-401f-48d6-bb77-c37a0585ea95\") " pod="openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl" Sep 08 00:54:15.250379 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.250238 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7lf4\" (UniqueName: \"kubernetes.io/projected/2d643ed9-65eb-4581-975d-df20e5f5407f-kube-api-access-c7lf4\") pod \"payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk\" (UID: \"2d643ed9-65eb-4581-975d-df20e5f5407f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk" Sep 08 00:54:15.250379 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.250298 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d643ed9-65eb-4581-975d-df20e5f5407f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk\" (UID: \"2d643ed9-65eb-4581-975d-df20e5f5407f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk" Sep 08 00:54:15.250467 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.250383 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlcj8\" (UniqueName: \"kubernetes.io/projected/9375e6a3-401f-48d6-bb77-c37a0585ea95-kube-api-access-nlcj8\") pod \"payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl\" (UID: \"9375e6a3-401f-48d6-bb77-c37a0585ea95\") " pod="openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl" Sep 08 00:54:15.351913 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.351872 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9375e6a3-401f-48d6-bb77-c37a0585ea95-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl\" (UID: \"9375e6a3-401f-48d6-bb77-c37a0585ea95\") " pod="openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl" Sep 08 00:54:15.352137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.351925 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7lf4\" (UniqueName: \"kubernetes.io/projected/2d643ed9-65eb-4581-975d-df20e5f5407f-kube-api-access-c7lf4\") pod \"payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk\" (UID: \"2d643ed9-65eb-4581-975d-df20e5f5407f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk" Sep 08 00:54:15.352137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.351972 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d643ed9-65eb-4581-975d-df20e5f5407f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk\" (UID: \"2d643ed9-65eb-4581-975d-df20e5f5407f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk" Sep 08 00:54:15.352137 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.352114 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlcj8\" (UniqueName: \"kubernetes.io/projected/9375e6a3-401f-48d6-bb77-c37a0585ea95-kube-api-access-nlcj8\") pod \"payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl\" (UID: \"9375e6a3-401f-48d6-bb77-c37a0585ea95\") " pod="openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl" Sep 08 00:54:15.352857 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.352829 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d643ed9-65eb-4581-975d-df20e5f5407f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk\" (UID: \"2d643ed9-65eb-4581-975d-df20e5f5407f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk" Sep 08 00:54:15.352973 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.352829 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9375e6a3-401f-48d6-bb77-c37a0585ea95-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl\" (UID: \"9375e6a3-401f-48d6-bb77-c37a0585ea95\") " pod="openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl" Sep 08 00:54:15.360604 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.360583 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7lf4\" (UniqueName: \"kubernetes.io/projected/2d643ed9-65eb-4581-975d-df20e5f5407f-kube-api-access-c7lf4\") pod \"payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk\" (UID: \"2d643ed9-65eb-4581-975d-df20e5f5407f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk" Sep 08 00:54:15.360797 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.360776 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlcj8\" (UniqueName: \"kubernetes.io/projected/9375e6a3-401f-48d6-bb77-c37a0585ea95-kube-api-access-nlcj8\") pod \"payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl\" (UID: \"9375e6a3-401f-48d6-bb77-c37a0585ea95\") " pod="openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl" Sep 08 00:54:15.465420 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.465318 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk" Sep 08 00:54:15.507916 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.507876 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl" Sep 08 00:54:15.739938 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.739857 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk"] Sep 08 00:54:15.742708 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:54:15.742680 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d643ed9_65eb_4581_975d_df20e5f5407f.slice/crio-ecf8571e29c46a7e659f576a92e4a07192da81d42da2b75ef2249e4e28db784f WatchSource:0}: Error finding container ecf8571e29c46a7e659f576a92e4a07192da81d42da2b75ef2249e4e28db784f: Status 404 returned error can't find the container with id ecf8571e29c46a7e659f576a92e4a07192da81d42da2b75ef2249e4e28db784f Sep 08 00:54:15.795754 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.795730 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl"] Sep 08 00:54:15.807138 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:54:15.807110 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9375e6a3_401f_48d6_bb77_c37a0585ea95.slice/crio-31c8cba2c096a31252b0cd4666697eb3d0ed1275ba6e80d2b1e847d87c331b95 WatchSource:0}: Error finding container 31c8cba2c096a31252b0cd4666697eb3d0ed1275ba6e80d2b1e847d87c331b95: Status 404 returned error can't find the container with id 31c8cba2c096a31252b0cd4666697eb3d0ed1275ba6e80d2b1e847d87c331b95 Sep 08 00:54:15.829949 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.829920 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl" event={"ID":"9375e6a3-401f-48d6-bb77-c37a0585ea95","Type":"ContainerStarted","Data":"31c8cba2c096a31252b0cd4666697eb3d0ed1275ba6e80d2b1e847d87c331b95"} Sep 08 00:54:15.831492 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.831471 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk" event={"ID":"2d643ed9-65eb-4581-975d-df20e5f5407f","Type":"ContainerStarted","Data":"bba50b7dd23d235ad281a9f675aa0c6b53cb9b5b3c73730ae08dc43e52596372"} Sep 08 00:54:15.831590 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.831507 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk" event={"ID":"2d643ed9-65eb-4581-975d-df20e5f5407f","Type":"ContainerStarted","Data":"ecf8571e29c46a7e659f576a92e4a07192da81d42da2b75ef2249e4e28db784f"} Sep 08 00:54:15.831590 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.831544 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk" Sep 08 00:54:15.848026 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:15.847961 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk" podStartSLOduration=0.847940571 podStartE2EDuration="847.940571ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:54:15.845756541 +0000 UTC m=+1269.540997701" watchObservedRunningTime="2026-09-08 00:54:15.847940571 +0000 UTC m=+1269.543182161" Sep 08 00:54:16.838492 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:16.838447 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl" event={"ID":"9375e6a3-401f-48d6-bb77-c37a0585ea95","Type":"ContainerStarted","Data":"092e19efb18f7112b34e7f1eae20f2bdc77d15cbabb509dd0025a482b4d0817a"} Sep 08 00:54:16.839017 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:16.838607 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl" Sep 08 00:54:16.855323 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:16.855275 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl" podStartSLOduration=1.8552596449999998 podStartE2EDuration="1.855259645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:54:16.8535689 +0000 UTC m=+1270.548810060" watchObservedRunningTime="2026-09-08 00:54:16.855259645 +0000 UTC m=+1270.550500821" Sep 08 00:54:26.841413 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:26.841383 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk" Sep 08 00:54:27.846640 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:27.846600 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl" Sep 08 00:54:40.957899 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:40.957865 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b1e9-lwrpj"] Sep 08 00:54:40.961767 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:40.961748 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b1e9-lwrpj" Sep 08 00:54:40.964051 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:40.964029 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 00:54:40.964184 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:40.964034 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dqw2\"" Sep 08 00:54:40.968030 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:40.968005 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b1e9-lwrpj"] Sep 08 00:54:40.975350 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:40.975314 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60431e60-1feb-4126-9689-cd0a02b2721f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-55b1e9-lwrpj\" (UID: \"60431e60-1feb-4126-9689-cd0a02b2721f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b1e9-lwrpj" Sep 08 00:54:41.076092 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:41.076058 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60431e60-1feb-4126-9689-cd0a02b2721f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-55b1e9-lwrpj\" (UID: \"60431e60-1feb-4126-9689-cd0a02b2721f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b1e9-lwrpj" Sep 08 00:54:41.076682 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:41.076665 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60431e60-1feb-4126-9689-cd0a02b2721f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-55b1e9-lwrpj\" (UID: \"60431e60-1feb-4126-9689-cd0a02b2721f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b1e9-lwrpj" Sep 08 00:54:41.275843 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:41.275756 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b1e9-lwrpj" Sep 08 00:54:41.543403 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:41.543345 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b1e9-lwrpj"] Sep 08 00:54:41.545084 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:54:41.545056 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60431e60_1feb_4126_9689_cd0a02b2721f.slice/crio-6dafcdf5e059d539caed0c1baf361d78ff1f2c80fefbbdb223e8a33cccd39e93 WatchSource:0}: Error finding container 6dafcdf5e059d539caed0c1baf361d78ff1f2c80fefbbdb223e8a33cccd39e93: Status 404 returned error can't find the container with id 6dafcdf5e059d539caed0c1baf361d78ff1f2c80fefbbdb223e8a33cccd39e93 Sep 08 00:54:41.967081 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:41.966976 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b1e9-lwrpj" event={"ID":"60431e60-1feb-4126-9689-cd0a02b2721f","Type":"ContainerStarted","Data":"79f63cab9b633b2e011dd805431b6fb5ee3441e3bd3a747f22576ef5d33af5f6"} Sep 08 00:54:41.967081 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:41.967037 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b1e9-lwrpj" event={"ID":"60431e60-1feb-4126-9689-cd0a02b2721f","Type":"ContainerStarted","Data":"6dafcdf5e059d539caed0c1baf361d78ff1f2c80fefbbdb223e8a33cccd39e93"} Sep 08 00:54:41.983527 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:41.983471 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b1e9-lwrpj" podStartSLOduration=1.9834555200000001 podStartE2EDuration="1.98345552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:54:41.981568638 +0000 UTC m=+1295.676809796" watchObservedRunningTime="2026-09-08 00:54:41.98345552 +0000 UTC m=+1295.678696719" Sep 08 00:54:42.973575 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:42.973536 2577 generic.go:356] "Generic (PLEG): container finished" podID="60431e60-1feb-4126-9689-cd0a02b2721f" containerID="79f63cab9b633b2e011dd805431b6fb5ee3441e3bd3a747f22576ef5d33af5f6" exitCode=0 Sep 08 00:54:42.973967 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:42.973575 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b1e9-lwrpj" event={"ID":"60431e60-1feb-4126-9689-cd0a02b2721f","Type":"ContainerDied","Data":"79f63cab9b633b2e011dd805431b6fb5ee3441e3bd3a747f22576ef5d33af5f6"} Sep 08 00:54:43.105719 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:43.105682 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b8c4de-btzhf"] Sep 08 00:54:43.109460 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:43.109434 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b8c4de-btzhf" Sep 08 00:54:43.116715 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:43.116676 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b8c4de-btzhf"] Sep 08 00:54:43.197192 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:43.197155 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e0cb6c-ef47-487e-a9d2-1af682746d3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b8c4de-btzhf\" (UID: \"19e0cb6c-ef47-487e-a9d2-1af682746d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b8c4de-btzhf" Sep 08 00:54:43.297781 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:43.297747 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e0cb6c-ef47-487e-a9d2-1af682746d3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b8c4de-btzhf\" (UID: \"19e0cb6c-ef47-487e-a9d2-1af682746d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b8c4de-btzhf" Sep 08 00:54:43.298404 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:43.298382 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e0cb6c-ef47-487e-a9d2-1af682746d3e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b8c4de-btzhf\" (UID: \"19e0cb6c-ef47-487e-a9d2-1af682746d3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b8c4de-btzhf" Sep 08 00:54:43.422743 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:43.422708 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b8c4de-btzhf" Sep 08 00:54:43.693903 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:43.693878 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b8c4de-btzhf"] Sep 08 00:54:43.695669 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:54:43.695640 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e0cb6c_ef47_487e_a9d2_1af682746d3e.slice/crio-7f3dada12cccec85efaf459ee898d4aa3e074ac6c7504e0d04416dc71f7c0b13 WatchSource:0}: Error finding container 7f3dada12cccec85efaf459ee898d4aa3e074ac6c7504e0d04416dc71f7c0b13: Status 404 returned error can't find the container with id 7f3dada12cccec85efaf459ee898d4aa3e074ac6c7504e0d04416dc71f7c0b13 Sep 08 00:54:43.981470 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:43.981392 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b8c4de-btzhf" event={"ID":"19e0cb6c-ef47-487e-a9d2-1af682746d3e","Type":"ContainerStarted","Data":"7f3dada12cccec85efaf459ee898d4aa3e074ac6c7504e0d04416dc71f7c0b13"} Sep 08 00:54:44.108026 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:44.107983 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b1e9-lwrpj" Sep 08 00:54:44.204739 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:44.204709 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60431e60-1feb-4126-9689-cd0a02b2721f-maas-api-service-ca\") pod \"60431e60-1feb-4126-9689-cd0a02b2721f\" (UID: \"60431e60-1feb-4126-9689-cd0a02b2721f\") " Sep 08 00:54:44.205064 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:44.205040 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60431e60-1feb-4126-9689-cd0a02b2721f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "60431e60-1feb-4126-9689-cd0a02b2721f" (UID: "60431e60-1feb-4126-9689-cd0a02b2721f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:54:44.305539 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:44.305504 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60431e60-1feb-4126-9689-cd0a02b2721f-maas-api-service-ca\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:54:44.989444 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:44.989411 2577 generic.go:356] "Generic (PLEG): container finished" podID="19e0cb6c-ef47-487e-a9d2-1af682746d3e" containerID="1853d1fe9874c6fee10acb06b49d77d3626261e0df8810a496438d39d378a929" exitCode=0 Sep 08 00:54:44.989876 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:44.989492 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b8c4de-btzhf" event={"ID":"19e0cb6c-ef47-487e-a9d2-1af682746d3e","Type":"ContainerDied","Data":"1853d1fe9874c6fee10acb06b49d77d3626261e0df8810a496438d39d378a929"} Sep 08 00:54:44.990687 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:44.990671 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b1e9-lwrpj" Sep 08 00:54:44.990763 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:44.990715 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b1e9-lwrpj" event={"ID":"60431e60-1feb-4126-9689-cd0a02b2721f","Type":"ContainerDied","Data":"6dafcdf5e059d539caed0c1baf361d78ff1f2c80fefbbdb223e8a33cccd39e93"} Sep 08 00:54:44.990763 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:44.990757 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dafcdf5e059d539caed0c1baf361d78ff1f2c80fefbbdb223e8a33cccd39e93" Sep 08 00:54:46.125032 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:46.125007 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b8c4de-btzhf" Sep 08 00:54:46.220374 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:46.220341 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e0cb6c-ef47-487e-a9d2-1af682746d3e-maas-api-service-ca\") pod \"19e0cb6c-ef47-487e-a9d2-1af682746d3e\" (UID: \"19e0cb6c-ef47-487e-a9d2-1af682746d3e\") " Sep 08 00:54:46.220692 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:46.220669 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19e0cb6c-ef47-487e-a9d2-1af682746d3e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "19e0cb6c-ef47-487e-a9d2-1af682746d3e" (UID: "19e0cb6c-ef47-487e-a9d2-1af682746d3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:54:46.321749 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:46.321716 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e0cb6c-ef47-487e-a9d2-1af682746d3e-maas-api-service-ca\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:54:47.007867 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:47.007834 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b8c4de-btzhf" Sep 08 00:54:47.007867 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:47.007847 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b8c4de-btzhf" event={"ID":"19e0cb6c-ef47-487e-a9d2-1af682746d3e","Type":"ContainerDied","Data":"7f3dada12cccec85efaf459ee898d4aa3e074ac6c7504e0d04416dc71f7c0b13"} Sep 08 00:54:47.008150 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:47.007926 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f3dada12cccec85efaf459ee898d4aa3e074ac6c7504e0d04416dc71f7c0b13" Sep 08 00:54:53.932601 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:53.932566 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq"] Sep 08 00:54:53.933150 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:53.932833 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq" podUID="45ed17de-8d86-403c-8d99-992f1c82a501" containerName="payload-processing" containerID="cri-o://b16458a423c14fa05cc0e77e9e8cf38c7aec65f76003ec8d6bd99737ec19cacd" gracePeriod=30 Sep 08 00:54:53.942456 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:53.942430 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9"] Sep 08 00:54:53.942686 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:53.942654 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9" podUID="962c714d-8126-4564-99bc-319dd04b4c10" containerName="payload-pre-processing" containerID="cri-o://36ec4ae815b2521b83abb9c642fb1858e47cfe2ec7155c7076240c494a7f428a" gracePeriod=30 Sep 08 00:54:54.049748 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:54.049722 2577 generic.go:356] "Generic (PLEG): container finished" podID="45ed17de-8d86-403c-8d99-992f1c82a501" containerID="b16458a423c14fa05cc0e77e9e8cf38c7aec65f76003ec8d6bd99737ec19cacd" exitCode=0 Sep 08 00:54:54.049818 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:54.049786 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq" event={"ID":"45ed17de-8d86-403c-8d99-992f1c82a501","Type":"ContainerDied","Data":"b16458a423c14fa05cc0e77e9e8cf38c7aec65f76003ec8d6bd99737ec19cacd"} Sep 08 00:54:54.097885 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:54.097848 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b8c4de-btzhf"] Sep 08 00:54:54.102306 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:54.102156 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b8c4de-btzhf"] Sep 08 00:54:54.226069 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:54.226046 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq" Sep 08 00:54:54.229129 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:54.229106 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9" Sep 08 00:54:54.292490 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:54.292465 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2cpm\" (UniqueName: \"kubernetes.io/projected/962c714d-8126-4564-99bc-319dd04b4c10-kube-api-access-f2cpm\") pod \"962c714d-8126-4564-99bc-319dd04b4c10\" (UID: \"962c714d-8126-4564-99bc-319dd04b4c10\") " Sep 08 00:54:54.292651 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:54.292497 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ed17de-8d86-403c-8d99-992f1c82a501-plugins-config-volume\") pod \"45ed17de-8d86-403c-8d99-992f1c82a501\" (UID: \"45ed17de-8d86-403c-8d99-992f1c82a501\") " Sep 08 00:54:54.292716 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:54.292700 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqhd7\" (UniqueName: \"kubernetes.io/projected/45ed17de-8d86-403c-8d99-992f1c82a501-kube-api-access-dqhd7\") pod \"45ed17de-8d86-403c-8d99-992f1c82a501\" (UID: \"45ed17de-8d86-403c-8d99-992f1c82a501\") " Sep 08 00:54:54.292775 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:54.292749 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/962c714d-8126-4564-99bc-319dd04b4c10-plugins-config-volume\") pod \"962c714d-8126-4564-99bc-319dd04b4c10\" (UID: \"962c714d-8126-4564-99bc-319dd04b4c10\") " Sep 08 00:54:54.292856 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:54.292835 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ed17de-8d86-403c-8d99-992f1c82a501-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "45ed17de-8d86-403c-8d99-992f1c82a501" (UID: "45ed17de-8d86-403c-8d99-992f1c82a501"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:54:54.293136 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:54.293055 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ed17de-8d86-403c-8d99-992f1c82a501-plugins-config-volume\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:54:54.293261 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:54.293239 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/962c714d-8126-4564-99bc-319dd04b4c10-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "962c714d-8126-4564-99bc-319dd04b4c10" (UID: "962c714d-8126-4564-99bc-319dd04b4c10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:54:54.294644 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:54.294625 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/962c714d-8126-4564-99bc-319dd04b4c10-kube-api-access-f2cpm" (OuterVolumeSpecName: "kube-api-access-f2cpm") pod "962c714d-8126-4564-99bc-319dd04b4c10" (UID: "962c714d-8126-4564-99bc-319dd04b4c10"). InnerVolumeSpecName "kube-api-access-f2cpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:54:54.294724 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:54.294710 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ed17de-8d86-403c-8d99-992f1c82a501-kube-api-access-dqhd7" (OuterVolumeSpecName: "kube-api-access-dqhd7") pod "45ed17de-8d86-403c-8d99-992f1c82a501" (UID: "45ed17de-8d86-403c-8d99-992f1c82a501"). InnerVolumeSpecName "kube-api-access-dqhd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:54:54.393517 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:54.393490 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqhd7\" (UniqueName: \"kubernetes.io/projected/45ed17de-8d86-403c-8d99-992f1c82a501-kube-api-access-dqhd7\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:54:54.393517 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:54.393515 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/962c714d-8126-4564-99bc-319dd04b4c10-plugins-config-volume\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:54:54.393517 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:54.393524 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2cpm\" (UniqueName: \"kubernetes.io/projected/962c714d-8126-4564-99bc-319dd04b4c10-kube-api-access-f2cpm\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:54:54.791830 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:54.791793 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19e0cb6c-ef47-487e-a9d2-1af682746d3e" path="/var/lib/kubelet/pods/19e0cb6c-ef47-487e-a9d2-1af682746d3e/volumes" Sep 08 00:54:55.057905 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:55.057814 2577 generic.go:356] "Generic (PLEG): container finished" podID="962c714d-8126-4564-99bc-319dd04b4c10" containerID="36ec4ae815b2521b83abb9c642fb1858e47cfe2ec7155c7076240c494a7f428a" exitCode=0 Sep 08 00:54:55.057905 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:55.057862 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9" Sep 08 00:54:55.057905 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:55.057893 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9" event={"ID":"962c714d-8126-4564-99bc-319dd04b4c10","Type":"ContainerDied","Data":"36ec4ae815b2521b83abb9c642fb1858e47cfe2ec7155c7076240c494a7f428a"} Sep 08 00:54:55.058457 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:55.057943 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9" event={"ID":"962c714d-8126-4564-99bc-319dd04b4c10","Type":"ContainerDied","Data":"87dc947dfe7489c0af2a9e47d3efcc6cc75406769616dee553e658587fde5abf"} Sep 08 00:54:55.058457 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:55.057959 2577 scope.go:117] "RemoveContainer" containerID="36ec4ae815b2521b83abb9c642fb1858e47cfe2ec7155c7076240c494a7f428a" Sep 08 00:54:55.059481 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:55.059452 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq" event={"ID":"45ed17de-8d86-403c-8d99-992f1c82a501","Type":"ContainerDied","Data":"707ceb227ce1ee6104e14a4aa44f69cccf76def6291f1fc2956879676ea8f721"} Sep 08 00:54:55.059605 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:55.059467 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq" Sep 08 00:54:55.067690 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:55.067672 2577 scope.go:117] "RemoveContainer" containerID="36ec4ae815b2521b83abb9c642fb1858e47cfe2ec7155c7076240c494a7f428a" Sep 08 00:54:55.067976 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:54:55.067958 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36ec4ae815b2521b83abb9c642fb1858e47cfe2ec7155c7076240c494a7f428a\": container with ID starting with 36ec4ae815b2521b83abb9c642fb1858e47cfe2ec7155c7076240c494a7f428a not found: ID does not exist" containerID="36ec4ae815b2521b83abb9c642fb1858e47cfe2ec7155c7076240c494a7f428a" Sep 08 00:54:55.068073 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:55.067984 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36ec4ae815b2521b83abb9c642fb1858e47cfe2ec7155c7076240c494a7f428a"} err="failed to get container status \"36ec4ae815b2521b83abb9c642fb1858e47cfe2ec7155c7076240c494a7f428a\": rpc error: code = NotFound desc = could not find container \"36ec4ae815b2521b83abb9c642fb1858e47cfe2ec7155c7076240c494a7f428a\": container with ID starting with 36ec4ae815b2521b83abb9c642fb1858e47cfe2ec7155c7076240c494a7f428a not found: ID does not exist" Sep 08 00:54:55.068073 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:55.068016 2577 scope.go:117] "RemoveContainer" containerID="b16458a423c14fa05cc0e77e9e8cf38c7aec65f76003ec8d6bd99737ec19cacd" Sep 08 00:54:55.079163 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:55.079131 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9"] Sep 08 00:54:55.083639 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:55.083616 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b8c4de-54755f6d6b-2hxb9"] Sep 08 00:54:55.096921 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:55.096890 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq"] Sep 08 00:54:55.102679 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:55.102651 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b8c4de-85859fdcbc-cxlhq"] Sep 08 00:54:56.802300 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:56.798349 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45ed17de-8d86-403c-8d99-992f1c82a501" path="/var/lib/kubelet/pods/45ed17de-8d86-403c-8d99-992f1c82a501/volumes" Sep 08 00:54:56.802300 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:56.798943 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="962c714d-8126-4564-99bc-319dd04b4c10" path="/var/lib/kubelet/pods/962c714d-8126-4564-99bc-319dd04b4c10/volumes" Sep 08 00:54:56.802300 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:56.799336 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm"] Sep 08 00:54:56.802300 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:56.799605 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm" podUID="5148709d-5baf-4a04-bf5a-a0a8b36edced" containerName="payload-processing" containerID="cri-o://312f86fbeab012a8f4abce2b6d0379bb4c4ab3947827793c28fddde3d3c26d3e" gracePeriod=30 Sep 08 00:54:56.937552 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:56.937519 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b1e9-lwrpj"] Sep 08 00:54:56.940253 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:56.940227 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b1e9-lwrpj"] Sep 08 00:54:57.065087 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:57.065022 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm" Sep 08 00:54:57.072164 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:57.072128 2577 generic.go:356] "Generic (PLEG): container finished" podID="5148709d-5baf-4a04-bf5a-a0a8b36edced" containerID="312f86fbeab012a8f4abce2b6d0379bb4c4ab3947827793c28fddde3d3c26d3e" exitCode=0 Sep 08 00:54:57.072164 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:57.072158 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm" Sep 08 00:54:57.072361 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:57.072209 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm" event={"ID":"5148709d-5baf-4a04-bf5a-a0a8b36edced","Type":"ContainerDied","Data":"312f86fbeab012a8f4abce2b6d0379bb4c4ab3947827793c28fddde3d3c26d3e"} Sep 08 00:54:57.072361 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:57.072267 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm" event={"ID":"5148709d-5baf-4a04-bf5a-a0a8b36edced","Type":"ContainerDied","Data":"30f4071b5c6f3914f3274c90a05c4e7e9872a89e56e3f76b759326ef1ac868d1"} Sep 08 00:54:57.072361 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:57.072297 2577 scope.go:117] "RemoveContainer" containerID="312f86fbeab012a8f4abce2b6d0379bb4c4ab3947827793c28fddde3d3c26d3e" Sep 08 00:54:57.081462 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:57.081445 2577 scope.go:117] "RemoveContainer" containerID="312f86fbeab012a8f4abce2b6d0379bb4c4ab3947827793c28fddde3d3c26d3e" Sep 08 00:54:57.081751 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:54:57.081716 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"312f86fbeab012a8f4abce2b6d0379bb4c4ab3947827793c28fddde3d3c26d3e\": container with ID starting with 312f86fbeab012a8f4abce2b6d0379bb4c4ab3947827793c28fddde3d3c26d3e not found: ID does not exist" containerID="312f86fbeab012a8f4abce2b6d0379bb4c4ab3947827793c28fddde3d3c26d3e" Sep 08 00:54:57.081751 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:57.081744 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"312f86fbeab012a8f4abce2b6d0379bb4c4ab3947827793c28fddde3d3c26d3e"} err="failed to get container status \"312f86fbeab012a8f4abce2b6d0379bb4c4ab3947827793c28fddde3d3c26d3e\": rpc error: code = NotFound desc = could not find container \"312f86fbeab012a8f4abce2b6d0379bb4c4ab3947827793c28fddde3d3c26d3e\": container with ID starting with 312f86fbeab012a8f4abce2b6d0379bb4c4ab3947827793c28fddde3d3c26d3e not found: ID does not exist" Sep 08 00:54:57.118039 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:57.117986 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ftft\" (UniqueName: \"kubernetes.io/projected/5148709d-5baf-4a04-bf5a-a0a8b36edced-kube-api-access-4ftft\") pod \"5148709d-5baf-4a04-bf5a-a0a8b36edced\" (UID: \"5148709d-5baf-4a04-bf5a-a0a8b36edced\") " Sep 08 00:54:57.118160 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:57.118085 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5148709d-5baf-4a04-bf5a-a0a8b36edced-plugins-config-volume\") pod \"5148709d-5baf-4a04-bf5a-a0a8b36edced\" (UID: \"5148709d-5baf-4a04-bf5a-a0a8b36edced\") " Sep 08 00:54:57.118418 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:57.118397 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5148709d-5baf-4a04-bf5a-a0a8b36edced-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5148709d-5baf-4a04-bf5a-a0a8b36edced" (UID: "5148709d-5baf-4a04-bf5a-a0a8b36edced"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:54:57.120549 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:57.120513 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5148709d-5baf-4a04-bf5a-a0a8b36edced-kube-api-access-4ftft" (OuterVolumeSpecName: "kube-api-access-4ftft") pod "5148709d-5baf-4a04-bf5a-a0a8b36edced" (UID: "5148709d-5baf-4a04-bf5a-a0a8b36edced"). InnerVolumeSpecName "kube-api-access-4ftft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:54:57.218836 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:57.218810 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5148709d-5baf-4a04-bf5a-a0a8b36edced-plugins-config-volume\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:54:57.218836 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:57.218835 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ftft\" (UniqueName: \"kubernetes.io/projected/5148709d-5baf-4a04-bf5a-a0a8b36edced-kube-api-access-4ftft\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:54:57.394552 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:57.394519 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm"] Sep 08 00:54:57.396037 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:57.396016 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-55b1e9-7c8db98669-rkwgm"] Sep 08 00:54:58.790208 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:58.790175 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5148709d-5baf-4a04-bf5a-a0a8b36edced" path="/var/lib/kubelet/pods/5148709d-5baf-4a04-bf5a-a0a8b36edced/volumes" Sep 08 00:54:58.790579 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:54:58.790502 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60431e60-1feb-4126-9689-cd0a02b2721f" path="/var/lib/kubelet/pods/60431e60-1feb-4126-9689-cd0a02b2721f/volumes" Sep 08 00:55:03.518921 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.518888 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e8800-9xkf6"] Sep 08 00:55:03.519304 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.519213 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60431e60-1feb-4126-9689-cd0a02b2721f" containerName="revoke-keys" Sep 08 00:55:03.519304 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.519225 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="60431e60-1feb-4126-9689-cd0a02b2721f" containerName="revoke-keys" Sep 08 00:55:03.519304 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.519237 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5148709d-5baf-4a04-bf5a-a0a8b36edced" containerName="payload-processing" Sep 08 00:55:03.519304 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.519242 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5148709d-5baf-4a04-bf5a-a0a8b36edced" containerName="payload-processing" Sep 08 00:55:03.519304 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.519259 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="962c714d-8126-4564-99bc-319dd04b4c10" containerName="payload-pre-processing" Sep 08 00:55:03.519304 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.519264 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="962c714d-8126-4564-99bc-319dd04b4c10" containerName="payload-pre-processing" Sep 08 00:55:03.519304 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.519270 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45ed17de-8d86-403c-8d99-992f1c82a501" containerName="payload-processing" Sep 08 00:55:03.519304 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.519275 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="45ed17de-8d86-403c-8d99-992f1c82a501" containerName="payload-processing" Sep 08 00:55:03.519304 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.519283 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19e0cb6c-ef47-487e-a9d2-1af682746d3e" containerName="revoke-keys" Sep 08 00:55:03.519304 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.519288 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="19e0cb6c-ef47-487e-a9d2-1af682746d3e" containerName="revoke-keys" Sep 08 00:55:03.519621 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.519341 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="45ed17de-8d86-403c-8d99-992f1c82a501" containerName="payload-processing" Sep 08 00:55:03.519621 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.519349 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="19e0cb6c-ef47-487e-a9d2-1af682746d3e" containerName="revoke-keys" Sep 08 00:55:03.519621 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.519355 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5148709d-5baf-4a04-bf5a-a0a8b36edced" containerName="payload-processing" Sep 08 00:55:03.519621 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.519361 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="962c714d-8126-4564-99bc-319dd04b4c10" containerName="payload-pre-processing" Sep 08 00:55:03.519621 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.519366 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="60431e60-1feb-4126-9689-cd0a02b2721f" containerName="revoke-keys" Sep 08 00:55:03.522508 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.522492 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e8800-9xkf6" Sep 08 00:55:03.524825 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.524805 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dqw2\"" Sep 08 00:55:03.524950 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.524881 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 00:55:03.528870 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.528849 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e8800-9xkf6"] Sep 08 00:55:03.675186 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.675153 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f25a7db-af15-48a8-b6a6-de586aae6f91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3e8800-9xkf6\" (UID: \"6f25a7db-af15-48a8-b6a6-de586aae6f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e8800-9xkf6" Sep 08 00:55:03.776115 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.776031 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f25a7db-af15-48a8-b6a6-de586aae6f91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3e8800-9xkf6\" (UID: \"6f25a7db-af15-48a8-b6a6-de586aae6f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e8800-9xkf6" Sep 08 00:55:03.776654 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.776633 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f25a7db-af15-48a8-b6a6-de586aae6f91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3e8800-9xkf6\" (UID: \"6f25a7db-af15-48a8-b6a6-de586aae6f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e8800-9xkf6" Sep 08 00:55:03.833836 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:03.833806 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e8800-9xkf6" Sep 08 00:55:04.111343 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:04.111318 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e8800-9xkf6"] Sep 08 00:55:04.113588 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:55:04.113562 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f25a7db_af15_48a8_b6a6_de586aae6f91.slice/crio-78393420dfb2c0481a5e86c6d6cefa99443d14075e354588312ae8a558623170 WatchSource:0}: Error finding container 78393420dfb2c0481a5e86c6d6cefa99443d14075e354588312ae8a558623170: Status 404 returned error can't find the container with id 78393420dfb2c0481a5e86c6d6cefa99443d14075e354588312ae8a558623170 Sep 08 00:55:05.114160 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:05.114120 2577 generic.go:356] "Generic (PLEG): container finished" podID="6f25a7db-af15-48a8-b6a6-de586aae6f91" containerID="115d27e29584579d89a489aa0e6d6bb777e19e79c2a0e0e3d9a3888bfdfae404" exitCode=0 Sep 08 00:55:05.114544 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:05.114159 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e8800-9xkf6" event={"ID":"6f25a7db-af15-48a8-b6a6-de586aae6f91","Type":"ContainerDied","Data":"115d27e29584579d89a489aa0e6d6bb777e19e79c2a0e0e3d9a3888bfdfae404"} Sep 08 00:55:05.114544 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:05.114210 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e8800-9xkf6" event={"ID":"6f25a7db-af15-48a8-b6a6-de586aae6f91","Type":"ContainerStarted","Data":"78393420dfb2c0481a5e86c6d6cefa99443d14075e354588312ae8a558623170"} Sep 08 00:55:06.243254 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:06.243231 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e8800-9xkf6" Sep 08 00:55:06.398301 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:06.398215 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f25a7db-af15-48a8-b6a6-de586aae6f91-maas-api-service-ca\") pod \"6f25a7db-af15-48a8-b6a6-de586aae6f91\" (UID: \"6f25a7db-af15-48a8-b6a6-de586aae6f91\") " Sep 08 00:55:06.398572 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:06.398548 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f25a7db-af15-48a8-b6a6-de586aae6f91-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6f25a7db-af15-48a8-b6a6-de586aae6f91" (UID: "6f25a7db-af15-48a8-b6a6-de586aae6f91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:55:06.499393 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:06.499354 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f25a7db-af15-48a8-b6a6-de586aae6f91-maas-api-service-ca\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:55:06.952220 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:06.952188 2577 scope.go:117] "RemoveContainer" containerID="ac3c06b3547ad4ba59f5be49340b91f0a0a2556dee632da681b253960b87a723" Sep 08 00:55:06.962430 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:06.962404 2577 scope.go:117] "RemoveContainer" containerID="9d69747ff51d0a42cd4f05e1d55c0dd54079de7ca771b5cfa47b7f697fd3e11b" Sep 08 00:55:07.125199 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:07.125112 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e8800-9xkf6" Sep 08 00:55:07.125199 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:07.125118 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e8800-9xkf6" event={"ID":"6f25a7db-af15-48a8-b6a6-de586aae6f91","Type":"ContainerDied","Data":"78393420dfb2c0481a5e86c6d6cefa99443d14075e354588312ae8a558623170"} Sep 08 00:55:07.125199 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:07.125190 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78393420dfb2c0481a5e86c6d6cefa99443d14075e354588312ae8a558623170" Sep 08 00:55:19.031723 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.031688 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl"] Sep 08 00:55:19.032204 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.031984 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl" podUID="9375e6a3-401f-48d6-bb77-c37a0585ea95" containerName="payload-processing" containerID="cri-o://092e19efb18f7112b34e7f1eae20f2bdc77d15cbabb509dd0025a482b4d0817a" gracePeriod=30 Sep 08 00:55:19.038147 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.038099 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk"] Sep 08 00:55:19.038366 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.038320 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk" podUID="2d643ed9-65eb-4581-975d-df20e5f5407f" containerName="payload-pre-processing" containerID="cri-o://bba50b7dd23d235ad281a9f675aa0c6b53cb9b5b3c73730ae08dc43e52596372" gracePeriod=30 Sep 08 00:55:19.190324 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.190274 2577 generic.go:356] "Generic (PLEG): container finished" podID="9375e6a3-401f-48d6-bb77-c37a0585ea95" containerID="092e19efb18f7112b34e7f1eae20f2bdc77d15cbabb509dd0025a482b4d0817a" exitCode=0 Sep 08 00:55:19.190519 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.190419 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl" event={"ID":"9375e6a3-401f-48d6-bb77-c37a0585ea95","Type":"ContainerDied","Data":"092e19efb18f7112b34e7f1eae20f2bdc77d15cbabb509dd0025a482b4d0817a"} Sep 08 00:55:19.192757 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.192728 2577 generic.go:356] "Generic (PLEG): container finished" podID="2d643ed9-65eb-4581-975d-df20e5f5407f" containerID="bba50b7dd23d235ad281a9f675aa0c6b53cb9b5b3c73730ae08dc43e52596372" exitCode=0 Sep 08 00:55:19.192876 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.192796 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk" event={"ID":"2d643ed9-65eb-4581-975d-df20e5f5407f","Type":"ContainerDied","Data":"bba50b7dd23d235ad281a9f675aa0c6b53cb9b5b3c73730ae08dc43e52596372"} Sep 08 00:55:19.201500 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.201476 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e8800-9xkf6"] Sep 08 00:55:19.204220 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.204198 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3e8800-9xkf6"] Sep 08 00:55:19.313701 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.313681 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl" Sep 08 00:55:19.316200 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.316185 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk" Sep 08 00:55:19.410492 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.410458 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9375e6a3-401f-48d6-bb77-c37a0585ea95-plugins-config-volume\") pod \"9375e6a3-401f-48d6-bb77-c37a0585ea95\" (UID: \"9375e6a3-401f-48d6-bb77-c37a0585ea95\") " Sep 08 00:55:19.410672 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.410531 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d643ed9-65eb-4581-975d-df20e5f5407f-plugins-config-volume\") pod \"2d643ed9-65eb-4581-975d-df20e5f5407f\" (UID: \"2d643ed9-65eb-4581-975d-df20e5f5407f\") " Sep 08 00:55:19.410672 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.410635 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlcj8\" (UniqueName: \"kubernetes.io/projected/9375e6a3-401f-48d6-bb77-c37a0585ea95-kube-api-access-nlcj8\") pod \"9375e6a3-401f-48d6-bb77-c37a0585ea95\" (UID: \"9375e6a3-401f-48d6-bb77-c37a0585ea95\") " Sep 08 00:55:19.410775 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.410682 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7lf4\" (UniqueName: \"kubernetes.io/projected/2d643ed9-65eb-4581-975d-df20e5f5407f-kube-api-access-c7lf4\") pod \"2d643ed9-65eb-4581-975d-df20e5f5407f\" (UID: \"2d643ed9-65eb-4581-975d-df20e5f5407f\") " Sep 08 00:55:19.410898 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.410874 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d643ed9-65eb-4581-975d-df20e5f5407f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2d643ed9-65eb-4581-975d-df20e5f5407f" (UID: "2d643ed9-65eb-4581-975d-df20e5f5407f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:55:19.410951 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.410878 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9375e6a3-401f-48d6-bb77-c37a0585ea95-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9375e6a3-401f-48d6-bb77-c37a0585ea95" (UID: "9375e6a3-401f-48d6-bb77-c37a0585ea95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:55:19.411009 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.410968 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9375e6a3-401f-48d6-bb77-c37a0585ea95-plugins-config-volume\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:55:19.411009 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.410982 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d643ed9-65eb-4581-975d-df20e5f5407f-plugins-config-volume\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:55:19.412695 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.412666 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9375e6a3-401f-48d6-bb77-c37a0585ea95-kube-api-access-nlcj8" (OuterVolumeSpecName: "kube-api-access-nlcj8") pod "9375e6a3-401f-48d6-bb77-c37a0585ea95" (UID: "9375e6a3-401f-48d6-bb77-c37a0585ea95"). InnerVolumeSpecName "kube-api-access-nlcj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:55:19.412803 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.412708 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d643ed9-65eb-4581-975d-df20e5f5407f-kube-api-access-c7lf4" (OuterVolumeSpecName: "kube-api-access-c7lf4") pod "2d643ed9-65eb-4581-975d-df20e5f5407f" (UID: "2d643ed9-65eb-4581-975d-df20e5f5407f"). InnerVolumeSpecName "kube-api-access-c7lf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:55:19.512374 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.512341 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlcj8\" (UniqueName: \"kubernetes.io/projected/9375e6a3-401f-48d6-bb77-c37a0585ea95-kube-api-access-nlcj8\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:55:19.512374 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:19.512369 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7lf4\" (UniqueName: \"kubernetes.io/projected/2d643ed9-65eb-4581-975d-df20e5f5407f-kube-api-access-c7lf4\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:55:20.198799 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:20.198768 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl" Sep 08 00:55:20.199308 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:20.198792 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl" event={"ID":"9375e6a3-401f-48d6-bb77-c37a0585ea95","Type":"ContainerDied","Data":"31c8cba2c096a31252b0cd4666697eb3d0ed1275ba6e80d2b1e847d87c331b95"} Sep 08 00:55:20.199308 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:20.198857 2577 scope.go:117] "RemoveContainer" containerID="092e19efb18f7112b34e7f1eae20f2bdc77d15cbabb509dd0025a482b4d0817a" Sep 08 00:55:20.200248 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:20.200074 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk" Sep 08 00:55:20.200248 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:20.200074 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk" event={"ID":"2d643ed9-65eb-4581-975d-df20e5f5407f","Type":"ContainerDied","Data":"ecf8571e29c46a7e659f576a92e4a07192da81d42da2b75ef2249e4e28db784f"} Sep 08 00:55:20.208149 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:20.207884 2577 scope.go:117] "RemoveContainer" containerID="bba50b7dd23d235ad281a9f675aa0c6b53cb9b5b3c73730ae08dc43e52596372" Sep 08 00:55:20.223255 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:20.223229 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk"] Sep 08 00:55:20.227167 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:20.227146 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3e8800-6d77c9ddd6-d9rkk"] Sep 08 00:55:20.236565 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:20.236545 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl"] Sep 08 00:55:20.240237 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:20.240218 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3e8800-6bf6f89449-vc9vl"] Sep 08 00:55:20.790713 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:20.790679 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d643ed9-65eb-4581-975d-df20e5f5407f" path="/var/lib/kubelet/pods/2d643ed9-65eb-4581-975d-df20e5f5407f/volumes" Sep 08 00:55:20.791027 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:20.791011 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f25a7db-af15-48a8-b6a6-de586aae6f91" path="/var/lib/kubelet/pods/6f25a7db-af15-48a8-b6a6-de586aae6f91/volumes" Sep 08 00:55:20.791343 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:20.791331 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9375e6a3-401f-48d6-bb77-c37a0585ea95" path="/var/lib/kubelet/pods/9375e6a3-401f-48d6-bb77-c37a0585ea95/volumes" Sep 08 00:55:25.752317 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:25.752279 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c99af4-qc6gb"] Sep 08 00:55:25.752798 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:25.752626 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d643ed9-65eb-4581-975d-df20e5f5407f" containerName="payload-pre-processing" Sep 08 00:55:25.752798 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:25.752638 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d643ed9-65eb-4581-975d-df20e5f5407f" containerName="payload-pre-processing" Sep 08 00:55:25.752798 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:25.752655 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9375e6a3-401f-48d6-bb77-c37a0585ea95" containerName="payload-processing" Sep 08 00:55:25.752798 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:25.752660 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9375e6a3-401f-48d6-bb77-c37a0585ea95" containerName="payload-processing" Sep 08 00:55:25.752798 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:25.752666 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f25a7db-af15-48a8-b6a6-de586aae6f91" containerName="revoke-keys" Sep 08 00:55:25.752798 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:25.752673 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f25a7db-af15-48a8-b6a6-de586aae6f91" containerName="revoke-keys" Sep 08 00:55:25.752798 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:25.752729 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f25a7db-af15-48a8-b6a6-de586aae6f91" containerName="revoke-keys" Sep 08 00:55:25.752798 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:25.752738 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d643ed9-65eb-4581-975d-df20e5f5407f" containerName="payload-pre-processing" Sep 08 00:55:25.752798 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:25.752744 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9375e6a3-401f-48d6-bb77-c37a0585ea95" containerName="payload-processing" Sep 08 00:55:25.756714 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:25.756697 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c99af4-qc6gb" Sep 08 00:55:25.759208 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:25.759192 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dqw2\"" Sep 08 00:55:25.759286 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:25.759263 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 00:55:25.762988 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:25.762962 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c99af4-qc6gb"] Sep 08 00:55:25.871252 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:25.871211 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d42bd025-0e05-4ad1-b342-e5da0b0bc694-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c99af4-qc6gb\" (UID: \"d42bd025-0e05-4ad1-b342-e5da0b0bc694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c99af4-qc6gb" Sep 08 00:55:25.972582 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:25.972544 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d42bd025-0e05-4ad1-b342-e5da0b0bc694-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c99af4-qc6gb\" (UID: \"d42bd025-0e05-4ad1-b342-e5da0b0bc694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c99af4-qc6gb" Sep 08 00:55:25.973240 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:25.973222 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d42bd025-0e05-4ad1-b342-e5da0b0bc694-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c99af4-qc6gb\" (UID: \"d42bd025-0e05-4ad1-b342-e5da0b0bc694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c99af4-qc6gb" Sep 08 00:55:26.068218 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:26.068188 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c99af4-qc6gb" Sep 08 00:55:26.339010 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:26.338935 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c99af4-qc6gb"] Sep 08 00:55:26.342720 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:55:26.342669 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42bd025_0e05_4ad1_b342_e5da0b0bc694.slice/crio-1af3576245fa94527a4ed1d5b5fb8d342b491a7499bff59073fd23257dc5ddbd WatchSource:0}: Error finding container 1af3576245fa94527a4ed1d5b5fb8d342b491a7499bff59073fd23257dc5ddbd: Status 404 returned error can't find the container with id 1af3576245fa94527a4ed1d5b5fb8d342b491a7499bff59073fd23257dc5ddbd Sep 08 00:55:27.238780 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:27.238744 2577 generic.go:356] "Generic (PLEG): container finished" podID="d42bd025-0e05-4ad1-b342-e5da0b0bc694" containerID="03b1f467754bf989216cd418371f7067023a20ceb97b98be77a627787269c4e5" exitCode=0 Sep 08 00:55:27.239165 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:27.238800 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c99af4-qc6gb" event={"ID":"d42bd025-0e05-4ad1-b342-e5da0b0bc694","Type":"ContainerDied","Data":"03b1f467754bf989216cd418371f7067023a20ceb97b98be77a627787269c4e5"} Sep 08 00:55:27.239165 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:27.238889 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c99af4-qc6gb" event={"ID":"d42bd025-0e05-4ad1-b342-e5da0b0bc694","Type":"ContainerStarted","Data":"1af3576245fa94527a4ed1d5b5fb8d342b491a7499bff59073fd23257dc5ddbd"} Sep 08 00:55:28.370752 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:28.370727 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c99af4-qc6gb" Sep 08 00:55:28.389953 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:28.389928 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d42bd025-0e05-4ad1-b342-e5da0b0bc694-maas-api-service-ca\") pod \"d42bd025-0e05-4ad1-b342-e5da0b0bc694\" (UID: \"d42bd025-0e05-4ad1-b342-e5da0b0bc694\") " Sep 08 00:55:28.390567 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:28.390534 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d42bd025-0e05-4ad1-b342-e5da0b0bc694-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d42bd025-0e05-4ad1-b342-e5da0b0bc694" (UID: "d42bd025-0e05-4ad1-b342-e5da0b0bc694"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:55:28.491191 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:28.491159 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d42bd025-0e05-4ad1-b342-e5da0b0bc694-maas-api-service-ca\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:55:29.249365 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:29.249253 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c99af4-qc6gb" Sep 08 00:55:29.249365 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:29.249268 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c99af4-qc6gb" event={"ID":"d42bd025-0e05-4ad1-b342-e5da0b0bc694","Type":"ContainerDied","Data":"1af3576245fa94527a4ed1d5b5fb8d342b491a7499bff59073fd23257dc5ddbd"} Sep 08 00:55:29.249365 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:29.249318 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1af3576245fa94527a4ed1d5b5fb8d342b491a7499bff59073fd23257dc5ddbd" Sep 08 00:55:36.507866 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:36.507828 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt"] Sep 08 00:55:36.508327 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:36.508101 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt" podUID="85f65be3-8dcc-4373-a477-a619f34dc41a" containerName="payload-processing" containerID="cri-o://783c470181aebaa1098ca61ec9c7bc7a7f7c15d90c3f8dc43a5a3c100afe1ac1" gracePeriod=30 Sep 08 00:55:36.662460 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:36.662427 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c99af4-qc6gb"] Sep 08 00:55:36.665105 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:36.665080 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c99af4-qc6gb"] Sep 08 00:55:36.749058 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:36.749034 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt" Sep 08 00:55:36.790774 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:36.790747 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d42bd025-0e05-4ad1-b342-e5da0b0bc694" path="/var/lib/kubelet/pods/d42bd025-0e05-4ad1-b342-e5da0b0bc694/volumes" Sep 08 00:55:36.852840 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:36.852804 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f65be3-8dcc-4373-a477-a619f34dc41a-plugins-config-volume\") pod \"85f65be3-8dcc-4373-a477-a619f34dc41a\" (UID: \"85f65be3-8dcc-4373-a477-a619f34dc41a\") " Sep 08 00:55:36.853029 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:36.852867 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8gf9\" (UniqueName: \"kubernetes.io/projected/85f65be3-8dcc-4373-a477-a619f34dc41a-kube-api-access-z8gf9\") pod \"85f65be3-8dcc-4373-a477-a619f34dc41a\" (UID: \"85f65be3-8dcc-4373-a477-a619f34dc41a\") " Sep 08 00:55:36.853215 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:36.853190 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f65be3-8dcc-4373-a477-a619f34dc41a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "85f65be3-8dcc-4373-a477-a619f34dc41a" (UID: "85f65be3-8dcc-4373-a477-a619f34dc41a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:55:36.854869 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:36.854854 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f65be3-8dcc-4373-a477-a619f34dc41a-kube-api-access-z8gf9" (OuterVolumeSpecName: "kube-api-access-z8gf9") pod "85f65be3-8dcc-4373-a477-a619f34dc41a" (UID: "85f65be3-8dcc-4373-a477-a619f34dc41a"). InnerVolumeSpecName "kube-api-access-z8gf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:55:36.954147 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:36.954118 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f65be3-8dcc-4373-a477-a619f34dc41a-plugins-config-volume\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:55:36.954147 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:36.954148 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8gf9\" (UniqueName: \"kubernetes.io/projected/85f65be3-8dcc-4373-a477-a619f34dc41a-kube-api-access-z8gf9\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:55:37.287678 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:37.287643 2577 generic.go:356] "Generic (PLEG): container finished" podID="85f65be3-8dcc-4373-a477-a619f34dc41a" containerID="783c470181aebaa1098ca61ec9c7bc7a7f7c15d90c3f8dc43a5a3c100afe1ac1" exitCode=0 Sep 08 00:55:37.287838 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:37.287688 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt" Sep 08 00:55:37.287838 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:37.287722 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt" event={"ID":"85f65be3-8dcc-4373-a477-a619f34dc41a","Type":"ContainerDied","Data":"783c470181aebaa1098ca61ec9c7bc7a7f7c15d90c3f8dc43a5a3c100afe1ac1"} Sep 08 00:55:37.287838 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:37.287772 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt" event={"ID":"85f65be3-8dcc-4373-a477-a619f34dc41a","Type":"ContainerDied","Data":"53d6caa4678543cd750235c105875c12af03ad4e9f30dc68d6c4cad483ab4c61"} Sep 08 00:55:37.287838 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:37.287795 2577 scope.go:117] "RemoveContainer" containerID="783c470181aebaa1098ca61ec9c7bc7a7f7c15d90c3f8dc43a5a3c100afe1ac1" Sep 08 00:55:37.297278 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:37.297261 2577 scope.go:117] "RemoveContainer" containerID="783c470181aebaa1098ca61ec9c7bc7a7f7c15d90c3f8dc43a5a3c100afe1ac1" Sep 08 00:55:37.297579 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:55:37.297560 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"783c470181aebaa1098ca61ec9c7bc7a7f7c15d90c3f8dc43a5a3c100afe1ac1\": container with ID starting with 783c470181aebaa1098ca61ec9c7bc7a7f7c15d90c3f8dc43a5a3c100afe1ac1 not found: ID does not exist" containerID="783c470181aebaa1098ca61ec9c7bc7a7f7c15d90c3f8dc43a5a3c100afe1ac1" Sep 08 00:55:37.297631 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:37.297588 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"783c470181aebaa1098ca61ec9c7bc7a7f7c15d90c3f8dc43a5a3c100afe1ac1"} err="failed to get container status \"783c470181aebaa1098ca61ec9c7bc7a7f7c15d90c3f8dc43a5a3c100afe1ac1\": rpc error: code = NotFound desc = could not find container \"783c470181aebaa1098ca61ec9c7bc7a7f7c15d90c3f8dc43a5a3c100afe1ac1\": container with ID starting with 783c470181aebaa1098ca61ec9c7bc7a7f7c15d90c3f8dc43a5a3c100afe1ac1 not found: ID does not exist" Sep 08 00:55:37.309705 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:37.309678 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt"] Sep 08 00:55:37.313962 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:37.313943 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-c99af4-558459dcdf-t4jwt"] Sep 08 00:55:38.789592 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:38.789560 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85f65be3-8dcc-4373-a477-a619f34dc41a" path="/var/lib/kubelet/pods/85f65be3-8dcc-4373-a477-a619f34dc41a/volumes" Sep 08 00:55:43.348903 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:43.348863 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f04ef7-v2m4x"] Sep 08 00:55:43.351344 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:43.349309 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f65be3-8dcc-4373-a477-a619f34dc41a" containerName="payload-processing" Sep 08 00:55:43.351344 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:43.349327 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f65be3-8dcc-4373-a477-a619f34dc41a" containerName="payload-processing" Sep 08 00:55:43.351344 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:43.349358 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d42bd025-0e05-4ad1-b342-e5da0b0bc694" containerName="revoke-keys" Sep 08 00:55:43.351344 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:43.349365 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d42bd025-0e05-4ad1-b342-e5da0b0bc694" containerName="revoke-keys" Sep 08 00:55:43.351344 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:43.349415 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="85f65be3-8dcc-4373-a477-a619f34dc41a" containerName="payload-processing" Sep 08 00:55:43.351344 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:43.349423 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d42bd025-0e05-4ad1-b342-e5da0b0bc694" containerName="revoke-keys" Sep 08 00:55:43.352190 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:43.352171 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f04ef7-v2m4x" Sep 08 00:55:43.354602 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:43.354576 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 00:55:43.354705 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:43.354583 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2dqw2\"" Sep 08 00:55:43.358820 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:43.358800 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f04ef7-v2m4x"] Sep 08 00:55:43.407898 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:43.407862 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769ebc33-e278-4e4f-8a24-dd1e17ee1405-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f04ef7-v2m4x\" (UID: \"769ebc33-e278-4e4f-8a24-dd1e17ee1405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f04ef7-v2m4x" Sep 08 00:55:43.509124 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:43.509095 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769ebc33-e278-4e4f-8a24-dd1e17ee1405-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f04ef7-v2m4x\" (UID: \"769ebc33-e278-4e4f-8a24-dd1e17ee1405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f04ef7-v2m4x" Sep 08 00:55:43.509717 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:43.509696 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769ebc33-e278-4e4f-8a24-dd1e17ee1405-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f04ef7-v2m4x\" (UID: \"769ebc33-e278-4e4f-8a24-dd1e17ee1405\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f04ef7-v2m4x" Sep 08 00:55:43.663853 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:43.663768 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f04ef7-v2m4x" Sep 08 00:55:43.944771 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:43.944698 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f04ef7-v2m4x"] Sep 08 00:55:43.947984 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:55:43.947955 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769ebc33_e278_4e4f_8a24_dd1e17ee1405.slice/crio-3801e836d9248b5f92d77be6f6c5f2802e7ad6f019920648011a6f50e6685d9e WatchSource:0}: Error finding container 3801e836d9248b5f92d77be6f6c5f2802e7ad6f019920648011a6f50e6685d9e: Status 404 returned error can't find the container with id 3801e836d9248b5f92d77be6f6c5f2802e7ad6f019920648011a6f50e6685d9e Sep 08 00:55:44.014786 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:44.014755 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck"] Sep 08 00:55:44.015051 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:44.015024 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" podUID="cb68d759-3712-4438-8ed8-19b3e331c7b5" containerName="istio-proxy" containerID="cri-o://592bf287fc6227350cdbf69d77b5b423806a95d28fd507b5c61fbdba6e60a927" gracePeriod=30 Sep 08 00:55:44.324161 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:44.324124 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f04ef7-v2m4x" event={"ID":"769ebc33-e278-4e4f-8a24-dd1e17ee1405","Type":"ContainerStarted","Data":"6f0692f2d2da8316f4bdf594d6d227c376007ab12d28d5e6b7fdc3d315674113"} Sep 08 00:55:44.324352 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:44.324212 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f04ef7-v2m4x" event={"ID":"769ebc33-e278-4e4f-8a24-dd1e17ee1405","Type":"ContainerStarted","Data":"3801e836d9248b5f92d77be6f6c5f2802e7ad6f019920648011a6f50e6685d9e"} Sep 08 00:55:44.341665 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:44.341598 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f04ef7-v2m4x" podStartSLOduration=1.34158288 podStartE2EDuration="1.34158288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:55:44.340418953 +0000 UTC m=+1358.035660113" watchObservedRunningTime="2026-09-08 00:55:44.34158288 +0000 UTC m=+1358.036824093" Sep 08 00:55:44.360030 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:44.359985 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e810-vtvbj"] Sep 08 00:55:44.363204 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:44.363188 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e810-vtvbj" Sep 08 00:55:44.370381 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:44.370354 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e810-vtvbj"] Sep 08 00:55:44.418799 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:44.418768 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9da4711-7d6e-4a23-8861-c90b4fb08c97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a8e810-vtvbj\" (UID: \"e9da4711-7d6e-4a23-8861-c90b4fb08c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e810-vtvbj" Sep 08 00:55:44.519862 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:44.519786 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9da4711-7d6e-4a23-8861-c90b4fb08c97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a8e810-vtvbj\" (UID: \"e9da4711-7d6e-4a23-8861-c90b4fb08c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e810-vtvbj" Sep 08 00:55:44.520387 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:44.520369 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9da4711-7d6e-4a23-8861-c90b4fb08c97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a8e810-vtvbj\" (UID: \"e9da4711-7d6e-4a23-8861-c90b4fb08c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e810-vtvbj" Sep 08 00:55:44.673953 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:44.673918 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e810-vtvbj" Sep 08 00:55:44.941882 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:44.941854 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e810-vtvbj"] Sep 08 00:55:44.943588 ip-10-0-139-137 kubenswrapper[2577]: W0908 00:55:44.943563 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9da4711_7d6e_4a23_8861_c90b4fb08c97.slice/crio-d33548a22dfdccc101782e6cb12a061aec97d5ae014252e5b5fc17d07a933c05 WatchSource:0}: Error finding container d33548a22dfdccc101782e6cb12a061aec97d5ae014252e5b5fc17d07a933c05: Status 404 returned error can't find the container with id d33548a22dfdccc101782e6cb12a061aec97d5ae014252e5b5fc17d07a933c05 Sep 08 00:55:45.330762 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:45.330729 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e810-vtvbj" event={"ID":"e9da4711-7d6e-4a23-8861-c90b4fb08c97","Type":"ContainerStarted","Data":"4d5fbcc775a04cd5c6961b15228add5401e3d33a4a495b32c49b1358ed24d835"} Sep 08 00:55:45.330944 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:45.330789 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e810-vtvbj" event={"ID":"e9da4711-7d6e-4a23-8861-c90b4fb08c97","Type":"ContainerStarted","Data":"d33548a22dfdccc101782e6cb12a061aec97d5ae014252e5b5fc17d07a933c05"} Sep 08 00:55:45.332253 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:45.332223 2577 generic.go:356] "Generic (PLEG): container finished" podID="769ebc33-e278-4e4f-8a24-dd1e17ee1405" containerID="6f0692f2d2da8316f4bdf594d6d227c376007ab12d28d5e6b7fdc3d315674113" exitCode=0 Sep 08 00:55:45.332381 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:45.332306 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f04ef7-v2m4x" event={"ID":"769ebc33-e278-4e4f-8a24-dd1e17ee1405","Type":"ContainerDied","Data":"6f0692f2d2da8316f4bdf594d6d227c376007ab12d28d5e6b7fdc3d315674113"} Sep 08 00:55:45.347030 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:45.346955 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e810-vtvbj" podStartSLOduration=1.346936367 podStartE2EDuration="1.346936367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 00:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 00:55:45.345254604 +0000 UTC m=+1359.040495764" watchObservedRunningTime="2026-09-08 00:55:45.346936367 +0000 UTC m=+1359.042177527" Sep 08 00:55:46.340182 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:46.340092 2577 generic.go:356] "Generic (PLEG): container finished" podID="e9da4711-7d6e-4a23-8861-c90b4fb08c97" containerID="4d5fbcc775a04cd5c6961b15228add5401e3d33a4a495b32c49b1358ed24d835" exitCode=0 Sep 08 00:55:46.340581 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:46.340185 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e810-vtvbj" event={"ID":"e9da4711-7d6e-4a23-8861-c90b4fb08c97","Type":"ContainerDied","Data":"4d5fbcc775a04cd5c6961b15228add5401e3d33a4a495b32c49b1358ed24d835"} Sep 08 00:55:46.470740 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:46.470717 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f04ef7-v2m4x" Sep 08 00:55:46.537121 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:46.537096 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769ebc33-e278-4e4f-8a24-dd1e17ee1405-maas-api-service-ca\") pod \"769ebc33-e278-4e4f-8a24-dd1e17ee1405\" (UID: \"769ebc33-e278-4e4f-8a24-dd1e17ee1405\") " Sep 08 00:55:46.537448 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:46.537425 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/769ebc33-e278-4e4f-8a24-dd1e17ee1405-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "769ebc33-e278-4e4f-8a24-dd1e17ee1405" (UID: "769ebc33-e278-4e4f-8a24-dd1e17ee1405"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:55:46.638032 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:46.637955 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769ebc33-e278-4e4f-8a24-dd1e17ee1405-maas-api-service-ca\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:55:47.351658 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:47.351624 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f04ef7-v2m4x" Sep 08 00:55:47.352211 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:47.351662 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f04ef7-v2m4x" event={"ID":"769ebc33-e278-4e4f-8a24-dd1e17ee1405","Type":"ContainerDied","Data":"3801e836d9248b5f92d77be6f6c5f2802e7ad6f019920648011a6f50e6685d9e"} Sep 08 00:55:47.352211 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:47.351735 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3801e836d9248b5f92d77be6f6c5f2802e7ad6f019920648011a6f50e6685d9e" Sep 08 00:55:47.361500 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:47.361459 2577 patch_prober.go:28] interesting pod/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 00:55:47.361646 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:47.361530 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" podUID="cb68d759-3712-4438-8ed8-19b3e331c7b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 00:55:47.474370 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:47.474346 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e810-vtvbj" Sep 08 00:55:47.544780 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:47.544741 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9da4711-7d6e-4a23-8861-c90b4fb08c97-maas-api-service-ca\") pod \"e9da4711-7d6e-4a23-8861-c90b4fb08c97\" (UID: \"e9da4711-7d6e-4a23-8861-c90b4fb08c97\") " Sep 08 00:55:47.545114 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:47.545092 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9da4711-7d6e-4a23-8861-c90b4fb08c97-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9da4711-7d6e-4a23-8861-c90b4fb08c97" (UID: "e9da4711-7d6e-4a23-8861-c90b4fb08c97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:55:47.646345 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:47.646267 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9da4711-7d6e-4a23-8861-c90b4fb08c97-maas-api-service-ca\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:55:48.357740 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:48.357692 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e810-vtvbj" event={"ID":"e9da4711-7d6e-4a23-8861-c90b4fb08c97","Type":"ContainerDied","Data":"d33548a22dfdccc101782e6cb12a061aec97d5ae014252e5b5fc17d07a933c05"} Sep 08 00:55:48.358155 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:48.357758 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d33548a22dfdccc101782e6cb12a061aec97d5ae014252e5b5fc17d07a933c05" Sep 08 00:55:48.358155 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:48.357704 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e810-vtvbj" Sep 08 00:55:49.259282 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.259260 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:55:49.361845 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.361811 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-workload-socket\") pod \"cb68d759-3712-4438-8ed8-19b3e331c7b5\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " Sep 08 00:55:49.362317 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.361863 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-credential-socket\") pod \"cb68d759-3712-4438-8ed8-19b3e331c7b5\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " Sep 08 00:55:49.362317 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.361920 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-podinfo\") pod \"cb68d759-3712-4438-8ed8-19b3e331c7b5\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " Sep 08 00:55:49.362317 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.361940 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-data\") pod \"cb68d759-3712-4438-8ed8-19b3e331c7b5\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " Sep 08 00:55:49.362317 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.361973 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-envoy\") pod \"cb68d759-3712-4438-8ed8-19b3e331c7b5\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " Sep 08 00:55:49.362317 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.362014 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-token\") pod \"cb68d759-3712-4438-8ed8-19b3e331c7b5\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " Sep 08 00:55:49.362317 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.362072 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-workload-certs\") pod \"cb68d759-3712-4438-8ed8-19b3e331c7b5\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " Sep 08 00:55:49.362317 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.362111 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb68d759-3712-4438-8ed8-19b3e331c7b5-istiod-ca-cert\") pod \"cb68d759-3712-4438-8ed8-19b3e331c7b5\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " Sep 08 00:55:49.362317 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.362114 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cb68d759-3712-4438-8ed8-19b3e331c7b5" (UID: "cb68d759-3712-4438-8ed8-19b3e331c7b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:55:49.362317 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.362136 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhvt7\" (UniqueName: \"kubernetes.io/projected/cb68d759-3712-4438-8ed8-19b3e331c7b5-kube-api-access-zhvt7\") pod \"cb68d759-3712-4438-8ed8-19b3e331c7b5\" (UID: \"cb68d759-3712-4438-8ed8-19b3e331c7b5\") " Sep 08 00:55:49.362317 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.362225 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cb68d759-3712-4438-8ed8-19b3e331c7b5" (UID: "cb68d759-3712-4438-8ed8-19b3e331c7b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:55:49.362870 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.362344 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-data" (OuterVolumeSpecName: "istio-data") pod "cb68d759-3712-4438-8ed8-19b3e331c7b5" (UID: "cb68d759-3712-4438-8ed8-19b3e331c7b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:55:49.362870 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.362368 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-workload-socket\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:55:49.362870 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.362400 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cb68d759-3712-4438-8ed8-19b3e331c7b5" (UID: "cb68d759-3712-4438-8ed8-19b3e331c7b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:55:49.362870 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.362416 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-credential-socket\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:55:49.362870 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.362630 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb68d759-3712-4438-8ed8-19b3e331c7b5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cb68d759-3712-4438-8ed8-19b3e331c7b5" (UID: "cb68d759-3712-4438-8ed8-19b3e331c7b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:55:49.364396 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.364358 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-token" (OuterVolumeSpecName: "istio-token") pod "cb68d759-3712-4438-8ed8-19b3e331c7b5" (UID: "cb68d759-3712-4438-8ed8-19b3e331c7b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:55:49.364582 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.364563 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cb68d759-3712-4438-8ed8-19b3e331c7b5" (UID: "cb68d759-3712-4438-8ed8-19b3e331c7b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 00:55:49.364668 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.364579 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cb68d759-3712-4438-8ed8-19b3e331c7b5" (UID: "cb68d759-3712-4438-8ed8-19b3e331c7b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:55:49.364668 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.364619 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb68d759-3712-4438-8ed8-19b3e331c7b5-kube-api-access-zhvt7" (OuterVolumeSpecName: "kube-api-access-zhvt7") pod "cb68d759-3712-4438-8ed8-19b3e331c7b5" (UID: "cb68d759-3712-4438-8ed8-19b3e331c7b5"). InnerVolumeSpecName "kube-api-access-zhvt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:55:49.365495 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.365470 2577 generic.go:356] "Generic (PLEG): container finished" podID="cb68d759-3712-4438-8ed8-19b3e331c7b5" containerID="592bf287fc6227350cdbf69d77b5b423806a95d28fd507b5c61fbdba6e60a927" exitCode=0 Sep 08 00:55:49.365603 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.365502 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" event={"ID":"cb68d759-3712-4438-8ed8-19b3e331c7b5","Type":"ContainerDied","Data":"592bf287fc6227350cdbf69d77b5b423806a95d28fd507b5c61fbdba6e60a927"} Sep 08 00:55:49.365603 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.365514 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" Sep 08 00:55:49.365603 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.365544 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck" event={"ID":"cb68d759-3712-4438-8ed8-19b3e331c7b5","Type":"ContainerDied","Data":"78f3a4643c455b10687b418e2dd9a98c421925950cac9895d2e599a51ecb971d"} Sep 08 00:55:49.365603 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.365567 2577 scope.go:117] "RemoveContainer" containerID="592bf287fc6227350cdbf69d77b5b423806a95d28fd507b5c61fbdba6e60a927" Sep 08 00:55:49.381045 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.381018 2577 scope.go:117] "RemoveContainer" containerID="24720273b54c3d0008ee8ecfdbbf7fadd15bc7077ff2a26b2328c34624b6aefc" Sep 08 00:55:49.388663 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.388647 2577 scope.go:117] "RemoveContainer" containerID="592bf287fc6227350cdbf69d77b5b423806a95d28fd507b5c61fbdba6e60a927" Sep 08 00:55:49.388924 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:55:49.388907 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"592bf287fc6227350cdbf69d77b5b423806a95d28fd507b5c61fbdba6e60a927\": container with ID starting with 592bf287fc6227350cdbf69d77b5b423806a95d28fd507b5c61fbdba6e60a927 not found: ID does not exist" containerID="592bf287fc6227350cdbf69d77b5b423806a95d28fd507b5c61fbdba6e60a927" Sep 08 00:55:49.388968 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.388931 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"592bf287fc6227350cdbf69d77b5b423806a95d28fd507b5c61fbdba6e60a927"} err="failed to get container status \"592bf287fc6227350cdbf69d77b5b423806a95d28fd507b5c61fbdba6e60a927\": rpc error: code = NotFound desc = could not find container \"592bf287fc6227350cdbf69d77b5b423806a95d28fd507b5c61fbdba6e60a927\": container with ID starting with 592bf287fc6227350cdbf69d77b5b423806a95d28fd507b5c61fbdba6e60a927 not found: ID does not exist" Sep 08 00:55:49.388968 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.388949 2577 scope.go:117] "RemoveContainer" containerID="24720273b54c3d0008ee8ecfdbbf7fadd15bc7077ff2a26b2328c34624b6aefc" Sep 08 00:55:49.389218 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:55:49.389202 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24720273b54c3d0008ee8ecfdbbf7fadd15bc7077ff2a26b2328c34624b6aefc\": container with ID starting with 24720273b54c3d0008ee8ecfdbbf7fadd15bc7077ff2a26b2328c34624b6aefc not found: ID does not exist" containerID="24720273b54c3d0008ee8ecfdbbf7fadd15bc7077ff2a26b2328c34624b6aefc" Sep 08 00:55:49.389285 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.389220 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24720273b54c3d0008ee8ecfdbbf7fadd15bc7077ff2a26b2328c34624b6aefc"} err="failed to get container status \"24720273b54c3d0008ee8ecfdbbf7fadd15bc7077ff2a26b2328c34624b6aefc\": rpc error: code = NotFound desc = could not find container \"24720273b54c3d0008ee8ecfdbbf7fadd15bc7077ff2a26b2328c34624b6aefc\": container with ID starting with 24720273b54c3d0008ee8ecfdbbf7fadd15bc7077ff2a26b2328c34624b6aefc not found: ID does not exist" Sep 08 00:55:49.391969 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.391949 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck"] Sep 08 00:55:49.395444 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.395425 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b8c4de-openshift-default-78557f796-f57ck"] Sep 08 00:55:49.463615 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.463590 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-podinfo\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:55:49.463615 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.463613 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-data\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:55:49.463783 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.463624 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-envoy\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:55:49.463783 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.463632 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb68d759-3712-4438-8ed8-19b3e331c7b5-istio-token\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:55:49.463783 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.463641 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb68d759-3712-4438-8ed8-19b3e331c7b5-workload-certs\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:55:49.463783 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.463649 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb68d759-3712-4438-8ed8-19b3e331c7b5-istiod-ca-cert\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:55:49.463783 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:49.463659 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhvt7\" (UniqueName: \"kubernetes.io/projected/cb68d759-3712-4438-8ed8-19b3e331c7b5-kube-api-access-zhvt7\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:55:50.790163 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:50.790132 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb68d759-3712-4438-8ed8-19b3e331c7b5" path="/var/lib/kubelet/pods/cb68d759-3712-4438-8ed8-19b3e331c7b5/volumes" Sep 08 00:55:54.393077 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:54.393044 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f04ef7-v2m4x"] Sep 08 00:55:54.396866 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:54.396841 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f04ef7-v2m4x"] Sep 08 00:55:54.790453 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:54.790421 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="769ebc33-e278-4e4f-8a24-dd1e17ee1405" path="/var/lib/kubelet/pods/769ebc33-e278-4e4f-8a24-dd1e17ee1405/volumes" Sep 08 00:55:55.344245 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:55.344208 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6"] Sep 08 00:55:55.344518 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:55.344482 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6" podUID="a43afd79-ff45-4dd6-b9f9-9fb355565a0c" containerName="payload-processing" containerID="cri-o://d8ccfde54b8b125e8396a389029533780cd1c4c973f02fc177f038e040f8e6d4" gracePeriod=30 Sep 08 00:55:55.506375 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:55.506336 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e810-vtvbj"] Sep 08 00:55:55.508307 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:55.508285 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a8e810-vtvbj"] Sep 08 00:55:55.586276 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:55.586252 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6" Sep 08 00:55:55.718226 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:55.718135 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v79vg\" (UniqueName: \"kubernetes.io/projected/a43afd79-ff45-4dd6-b9f9-9fb355565a0c-kube-api-access-v79vg\") pod \"a43afd79-ff45-4dd6-b9f9-9fb355565a0c\" (UID: \"a43afd79-ff45-4dd6-b9f9-9fb355565a0c\") " Sep 08 00:55:55.718226 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:55.718175 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43afd79-ff45-4dd6-b9f9-9fb355565a0c-plugins-config-volume\") pod \"a43afd79-ff45-4dd6-b9f9-9fb355565a0c\" (UID: \"a43afd79-ff45-4dd6-b9f9-9fb355565a0c\") " Sep 08 00:55:55.718566 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:55.718543 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a43afd79-ff45-4dd6-b9f9-9fb355565a0c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a43afd79-ff45-4dd6-b9f9-9fb355565a0c" (UID: "a43afd79-ff45-4dd6-b9f9-9fb355565a0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:55:55.720235 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:55.720214 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43afd79-ff45-4dd6-b9f9-9fb355565a0c-kube-api-access-v79vg" (OuterVolumeSpecName: "kube-api-access-v79vg") pod "a43afd79-ff45-4dd6-b9f9-9fb355565a0c" (UID: "a43afd79-ff45-4dd6-b9f9-9fb355565a0c"). InnerVolumeSpecName "kube-api-access-v79vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:55:55.819091 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:55.819043 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v79vg\" (UniqueName: \"kubernetes.io/projected/a43afd79-ff45-4dd6-b9f9-9fb355565a0c-kube-api-access-v79vg\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:55:55.819091 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:55.819084 2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a43afd79-ff45-4dd6-b9f9-9fb355565a0c-plugins-config-volume\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:55:56.400391 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:56.400355 2577 generic.go:356] "Generic (PLEG): container finished" podID="a43afd79-ff45-4dd6-b9f9-9fb355565a0c" containerID="d8ccfde54b8b125e8396a389029533780cd1c4c973f02fc177f038e040f8e6d4" exitCode=0 Sep 08 00:55:56.400582 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:56.400405 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6" Sep 08 00:55:56.400582 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:56.400444 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6" event={"ID":"a43afd79-ff45-4dd6-b9f9-9fb355565a0c","Type":"ContainerDied","Data":"d8ccfde54b8b125e8396a389029533780cd1c4c973f02fc177f038e040f8e6d4"} Sep 08 00:55:56.400582 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:56.400497 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6" event={"ID":"a43afd79-ff45-4dd6-b9f9-9fb355565a0c","Type":"ContainerDied","Data":"628dfdb79448853ec8e0ff2a31bfa0c97f9c80251088e9731b1ef1d365ec969c"} Sep 08 00:55:56.400582 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:56.400521 2577 scope.go:117] "RemoveContainer" containerID="d8ccfde54b8b125e8396a389029533780cd1c4c973f02fc177f038e040f8e6d4" Sep 08 00:55:56.409450 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:56.409435 2577 scope.go:117] "RemoveContainer" containerID="d8ccfde54b8b125e8396a389029533780cd1c4c973f02fc177f038e040f8e6d4" Sep 08 00:55:56.409730 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:55:56.409709 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8ccfde54b8b125e8396a389029533780cd1c4c973f02fc177f038e040f8e6d4\": container with ID starting with d8ccfde54b8b125e8396a389029533780cd1c4c973f02fc177f038e040f8e6d4 not found: ID does not exist" containerID="d8ccfde54b8b125e8396a389029533780cd1c4c973f02fc177f038e040f8e6d4" Sep 08 00:55:56.409778 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:56.409739 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8ccfde54b8b125e8396a389029533780cd1c4c973f02fc177f038e040f8e6d4"} err="failed to get container status \"d8ccfde54b8b125e8396a389029533780cd1c4c973f02fc177f038e040f8e6d4\": rpc error: code = NotFound desc = could not find container \"d8ccfde54b8b125e8396a389029533780cd1c4c973f02fc177f038e040f8e6d4\": container with ID starting with d8ccfde54b8b125e8396a389029533780cd1c4c973f02fc177f038e040f8e6d4 not found: ID does not exist" Sep 08 00:55:56.421734 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:56.421709 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6"] Sep 08 00:55:56.425320 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:56.425302 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-a8e810-7667dfdcf5-dnfp6"] Sep 08 00:55:56.789763 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:56.789729 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a43afd79-ff45-4dd6-b9f9-9fb355565a0c" path="/var/lib/kubelet/pods/a43afd79-ff45-4dd6-b9f9-9fb355565a0c/volumes" Sep 08 00:55:56.790220 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:55:56.790053 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9da4711-7d6e-4a23-8861-c90b4fb08c97" path="/var/lib/kubelet/pods/e9da4711-7d6e-4a23-8861-c90b4fb08c97/volumes" Sep 08 00:56:00.767442 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:00.767409 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq"] Sep 08 00:56:00.767874 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:00.767627 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" podUID="1fb6451f-d144-45b9-a925-bf63f81ff56f" containerName="istio-proxy" containerID="cri-o://9ba5a7bd685c5512fc81b16622116df85753a41f5425ebf1a9e35802f1df0305" gracePeriod=30 Sep 08 00:56:01.081259 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:01.081221 2577 patch_prober.go:28] interesting pod/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 00:56:01.081437 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:01.081291 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" podUID="1fb6451f-d144-45b9-a925-bf63f81ff56f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 00:56:06.010780 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.010756 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:56:06.107217 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.107144 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-workload-certs\") pod \"1fb6451f-d144-45b9-a925-bf63f81ff56f\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " Sep 08 00:56:06.107217 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.107204 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb6451f-d144-45b9-a925-bf63f81ff56f-istiod-ca-cert\") pod \"1fb6451f-d144-45b9-a925-bf63f81ff56f\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " Sep 08 00:56:06.107439 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.107231 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-token\") pod \"1fb6451f-d144-45b9-a925-bf63f81ff56f\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " Sep 08 00:56:06.107439 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.107251 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-credential-socket\") pod \"1fb6451f-d144-45b9-a925-bf63f81ff56f\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " Sep 08 00:56:06.107439 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.107287 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-envoy\") pod \"1fb6451f-d144-45b9-a925-bf63f81ff56f\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " Sep 08 00:56:06.107439 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.107311 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-data\") pod \"1fb6451f-d144-45b9-a925-bf63f81ff56f\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " Sep 08 00:56:06.107439 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.107350 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-www2v\" (UniqueName: \"kubernetes.io/projected/1fb6451f-d144-45b9-a925-bf63f81ff56f-kube-api-access-www2v\") pod \"1fb6451f-d144-45b9-a925-bf63f81ff56f\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " Sep 08 00:56:06.107439 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.107383 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-workload-socket\") pod \"1fb6451f-d144-45b9-a925-bf63f81ff56f\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " Sep 08 00:56:06.107439 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.107415 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-podinfo\") pod \"1fb6451f-d144-45b9-a925-bf63f81ff56f\" (UID: \"1fb6451f-d144-45b9-a925-bf63f81ff56f\") " Sep 08 00:56:06.107439 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.107422 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1fb6451f-d144-45b9-a925-bf63f81ff56f" (UID: "1fb6451f-d144-45b9-a925-bf63f81ff56f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:56:06.107824 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.107503 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1fb6451f-d144-45b9-a925-bf63f81ff56f" (UID: "1fb6451f-d144-45b9-a925-bf63f81ff56f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:56:06.107824 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.107657 2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-workload-certs\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:56:06.107824 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.107677 2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-credential-socket\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:56:06.107824 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.107681 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-data" (OuterVolumeSpecName: "istio-data") pod "1fb6451f-d144-45b9-a925-bf63f81ff56f" (UID: "1fb6451f-d144-45b9-a925-bf63f81ff56f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:56:06.107824 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.107729 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb6451f-d144-45b9-a925-bf63f81ff56f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1fb6451f-d144-45b9-a925-bf63f81ff56f" (UID: "1fb6451f-d144-45b9-a925-bf63f81ff56f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 00:56:06.107824 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.107750 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1fb6451f-d144-45b9-a925-bf63f81ff56f" (UID: "1fb6451f-d144-45b9-a925-bf63f81ff56f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:56:06.109447 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.109421 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1fb6451f-d144-45b9-a925-bf63f81ff56f" (UID: "1fb6451f-d144-45b9-a925-bf63f81ff56f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 00:56:06.109562 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.109507 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb6451f-d144-45b9-a925-bf63f81ff56f-kube-api-access-www2v" (OuterVolumeSpecName: "kube-api-access-www2v") pod "1fb6451f-d144-45b9-a925-bf63f81ff56f" (UID: "1fb6451f-d144-45b9-a925-bf63f81ff56f"). InnerVolumeSpecName "kube-api-access-www2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:56:06.109562 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.109540 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-token" (OuterVolumeSpecName: "istio-token") pod "1fb6451f-d144-45b9-a925-bf63f81ff56f" (UID: "1fb6451f-d144-45b9-a925-bf63f81ff56f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 00:56:06.109635 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.109617 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1fb6451f-d144-45b9-a925-bf63f81ff56f" (UID: "1fb6451f-d144-45b9-a925-bf63f81ff56f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 08 00:56:06.208318 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.208282 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-token\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:56:06.208318 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.208310 2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-envoy\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:56:06.208318 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.208319 2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-data\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:56:06.208571 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.208329 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-www2v\" (UniqueName: \"kubernetes.io/projected/1fb6451f-d144-45b9-a925-bf63f81ff56f-kube-api-access-www2v\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:56:06.208571 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.208339 2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb6451f-d144-45b9-a925-bf63f81ff56f-workload-socket\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:56:06.208571 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.208347 2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb6451f-d144-45b9-a925-bf63f81ff56f-istio-podinfo\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:56:06.208571 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.208357 2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb6451f-d144-45b9-a925-bf63f81ff56f-istiod-ca-cert\") on node \"ip-10-0-139-137.ec2.internal\" DevicePath \"\"" Sep 08 00:56:06.461096 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.460983 2577 generic.go:356] "Generic (PLEG): container finished" podID="1fb6451f-d144-45b9-a925-bf63f81ff56f" containerID="9ba5a7bd685c5512fc81b16622116df85753a41f5425ebf1a9e35802f1df0305" exitCode=0 Sep 08 00:56:06.461096 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.461051 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" Sep 08 00:56:06.461096 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.461066 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" event={"ID":"1fb6451f-d144-45b9-a925-bf63f81ff56f","Type":"ContainerDied","Data":"9ba5a7bd685c5512fc81b16622116df85753a41f5425ebf1a9e35802f1df0305"} Sep 08 00:56:06.461389 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.461133 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq" event={"ID":"1fb6451f-d144-45b9-a925-bf63f81ff56f","Type":"ContainerDied","Data":"21287d37f455411ab38609299f42816af3de55e707f1506938d9952a8a117c38"} Sep 08 00:56:06.461389 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.461164 2577 scope.go:117] "RemoveContainer" containerID="9ba5a7bd685c5512fc81b16622116df85753a41f5425ebf1a9e35802f1df0305" Sep 08 00:56:06.471057 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.471039 2577 scope.go:117] "RemoveContainer" containerID="2aeba9c454e9e80b940e79321d5233671792733bf354ce26a533c81f37cd0384" Sep 08 00:56:06.478848 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.478831 2577 scope.go:117] "RemoveContainer" containerID="9ba5a7bd685c5512fc81b16622116df85753a41f5425ebf1a9e35802f1df0305" Sep 08 00:56:06.479116 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:56:06.479098 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba5a7bd685c5512fc81b16622116df85753a41f5425ebf1a9e35802f1df0305\": container with ID starting with 9ba5a7bd685c5512fc81b16622116df85753a41f5425ebf1a9e35802f1df0305 not found: ID does not exist" containerID="9ba5a7bd685c5512fc81b16622116df85753a41f5425ebf1a9e35802f1df0305" Sep 08 00:56:06.479163 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.479124 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba5a7bd685c5512fc81b16622116df85753a41f5425ebf1a9e35802f1df0305"} err="failed to get container status \"9ba5a7bd685c5512fc81b16622116df85753a41f5425ebf1a9e35802f1df0305\": rpc error: code = NotFound desc = could not find container \"9ba5a7bd685c5512fc81b16622116df85753a41f5425ebf1a9e35802f1df0305\": container with ID starting with 9ba5a7bd685c5512fc81b16622116df85753a41f5425ebf1a9e35802f1df0305 not found: ID does not exist" Sep 08 00:56:06.479163 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.479141 2577 scope.go:117] "RemoveContainer" containerID="2aeba9c454e9e80b940e79321d5233671792733bf354ce26a533c81f37cd0384" Sep 08 00:56:06.479383 ip-10-0-139-137 kubenswrapper[2577]: E0908 00:56:06.479365 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aeba9c454e9e80b940e79321d5233671792733bf354ce26a533c81f37cd0384\": container with ID starting with 2aeba9c454e9e80b940e79321d5233671792733bf354ce26a533c81f37cd0384 not found: ID does not exist" containerID="2aeba9c454e9e80b940e79321d5233671792733bf354ce26a533c81f37cd0384" Sep 08 00:56:06.479421 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.479390 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aeba9c454e9e80b940e79321d5233671792733bf354ce26a533c81f37cd0384"} err="failed to get container status \"2aeba9c454e9e80b940e79321d5233671792733bf354ce26a533c81f37cd0384\": rpc error: code = NotFound desc = could not find container \"2aeba9c454e9e80b940e79321d5233671792733bf354ce26a533c81f37cd0384\": container with ID starting with 2aeba9c454e9e80b940e79321d5233671792733bf354ce26a533c81f37cd0384 not found: ID does not exist" Sep 08 00:56:06.483809 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.483790 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq"] Sep 08 00:56:06.487603 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.487584 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-f04ef7-openshift-default-5d76c7b5b-m55dq"] Sep 08 00:56:06.790699 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.790668 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fb6451f-d144-45b9-a925-bf63f81ff56f" path="/var/lib/kubelet/pods/1fb6451f-d144-45b9-a925-bf63f81ff56f/volumes" Sep 08 00:56:06.995988 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:06.995958 2577 scope.go:117] "RemoveContainer" containerID="d47269076ab678da5a53be97b9b171ee84f5f462f4b15115bbd16a5f622fe1e9" Sep 08 00:56:07.003854 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:07.003836 2577 scope.go:117] "RemoveContainer" containerID="44bedde2e658059584ba9a49936814b991327bef482023da6138634ac3af07d0" Sep 08 00:56:07.010548 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:56:07.010529 2577 scope.go:117] "RemoveContainer" containerID="55503a902fa3d23b46abcee23454ebd643c89811346406e69de0829ad2ba8c43" Sep 08 00:58:06.786311 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:58:06.786269 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9nb9x_a2e160ae-d75d-4959-a5ab-d238b4113736/console-operator/1.log" Sep 08 00:58:06.790384 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:58:06.790358 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9nb9x_a2e160ae-d75d-4959-a5ab-d238b4113736/console-operator/1.log" Sep 08 00:58:07.061568 ip-10-0-139-137 kubenswrapper[2577]: I0908 00:58:07.061484 2577 scope.go:117] "RemoveContainer" containerID="ac1a9ad99838e5d37dd08886847e0268abe43695d319b719e6559b0e517726c6" Sep 08 01:01:07.094439 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:01:07.094408 2577 scope.go:117] "RemoveContainer" containerID="1853d1fe9874c6fee10acb06b49d77d3626261e0df8810a496438d39d378a929" Sep 08 01:01:07.102790 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:01:07.102768 2577 scope.go:117] "RemoveContainer" containerID="115d27e29584579d89a489aa0e6d6bb777e19e79c2a0e0e3d9a3888bfdfae404" Sep 08 01:01:07.109800 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:01:07.109782 2577 scope.go:117] "RemoveContainer" containerID="79f63cab9b633b2e011dd805431b6fb5ee3441e3bd3a747f22576ef5d33af5f6" Sep 08 01:02:07.136451 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:07.136421 2577 scope.go:117] "RemoveContainer" containerID="6f0692f2d2da8316f4bdf594d6d227c376007ab12d28d5e6b7fdc3d315674113" Sep 08 01:02:07.144305 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:07.144286 2577 scope.go:117] "RemoveContainer" containerID="4d5fbcc775a04cd5c6961b15228add5401e3d33a4a495b32c49b1358ed24d835" Sep 08 01:02:07.151070 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:07.151047 2577 scope.go:117] "RemoveContainer" containerID="03b1f467754bf989216cd418371f7067023a20ceb97b98be77a627787269c4e5" Sep 08 01:02:47.448248 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:47.448215 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29813805-zpx59_c2b3a14e-862b-408e-aa2d-e176e0bb7147/cleanup/0.log" Sep 08 01:02:49.519446 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:49.519416 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-h2gwx_14d4c77a-4999-4d40-803a-80b4f77aad5d/discovery/0.log" Sep 08 01:02:49.730228 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:49.730199 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58c5b795c4-npd8z_fbf31006-fc5d-4957-9766-b89b65cee90a/kube-auth-proxy/0.log" Sep 08 01:02:51.163134 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:51.163103 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-89945f574-tbcrg_e946943c-5a95-4267-b0bb-7b73f74f1ef9/main/0.log" Sep 08 01:02:51.272134 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:51.272100 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5757f658bb-hnnd8_9f89ca12-4299-43a7-950d-0d9ef5a0227b/main/0.log" Sep 08 01:02:56.569716 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.569676 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2vwnw/must-gather-xpbmv"] Sep 08 01:02:56.570218 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.570088 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="769ebc33-e278-4e4f-8a24-dd1e17ee1405" containerName="revoke-keys" Sep 08 01:02:56.570218 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.570105 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="769ebc33-e278-4e4f-8a24-dd1e17ee1405" containerName="revoke-keys" Sep 08 01:02:56.570218 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.570120 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb68d759-3712-4438-8ed8-19b3e331c7b5" containerName="istio-proxy" Sep 08 01:02:56.570218 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.570128 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb68d759-3712-4438-8ed8-19b3e331c7b5" containerName="istio-proxy" Sep 08 01:02:56.570218 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.570144 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a43afd79-ff45-4dd6-b9f9-9fb355565a0c" containerName="payload-processing" Sep 08 01:02:56.570218 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.570154 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a43afd79-ff45-4dd6-b9f9-9fb355565a0c" containerName="payload-processing" Sep 08 01:02:56.570218 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.570163 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9da4711-7d6e-4a23-8861-c90b4fb08c97" containerName="revoke-keys" Sep 08 01:02:56.570218 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.570170 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9da4711-7d6e-4a23-8861-c90b4fb08c97" containerName="revoke-keys" Sep 08 01:02:56.570218 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.570186 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb68d759-3712-4438-8ed8-19b3e331c7b5" containerName="istio-proxy" Sep 08 01:02:56.570218 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.570194 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb68d759-3712-4438-8ed8-19b3e331c7b5" containerName="istio-proxy" Sep 08 01:02:56.570218 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.570212 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fb6451f-d144-45b9-a925-bf63f81ff56f" containerName="istio-proxy" Sep 08 01:02:56.570218 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.570220 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fb6451f-d144-45b9-a925-bf63f81ff56f" containerName="istio-proxy" Sep 08 01:02:56.570821 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.570230 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fb6451f-d144-45b9-a925-bf63f81ff56f" containerName="istio-proxy" Sep 08 01:02:56.570821 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.570238 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fb6451f-d144-45b9-a925-bf63f81ff56f" containerName="istio-proxy" Sep 08 01:02:56.570821 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.570318 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fb6451f-d144-45b9-a925-bf63f81ff56f" containerName="istio-proxy" Sep 08 01:02:56.570821 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.570330 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb68d759-3712-4438-8ed8-19b3e331c7b5" containerName="istio-proxy" Sep 08 01:02:56.570821 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.570343 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a43afd79-ff45-4dd6-b9f9-9fb355565a0c" containerName="payload-processing" Sep 08 01:02:56.570821 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.570352 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="769ebc33-e278-4e4f-8a24-dd1e17ee1405" containerName="revoke-keys" Sep 08 01:02:56.570821 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.570361 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9da4711-7d6e-4a23-8861-c90b4fb08c97" containerName="revoke-keys" Sep 08 01:02:56.570821 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.570516 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fb6451f-d144-45b9-a925-bf63f81ff56f" containerName="istio-proxy" Sep 08 01:02:56.570821 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.570525 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb68d759-3712-4438-8ed8-19b3e331c7b5" containerName="istio-proxy" Sep 08 01:02:56.572884 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.572864 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vwnw/must-gather-xpbmv" Sep 08 01:02:56.575449 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.575423 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2vwnw\"/\"default-dockercfg-ptf2h\"" Sep 08 01:02:56.575967 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.575949 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2vwnw\"/\"kube-root-ca.crt\"" Sep 08 01:02:56.576080 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.575950 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2vwnw\"/\"openshift-service-ca.crt\"" Sep 08 01:02:56.584435 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.584413 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vwnw/must-gather-xpbmv"] Sep 08 01:02:56.601705 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.601679 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nmj6\" (UniqueName: \"kubernetes.io/projected/0b1a819f-4d2f-4ed6-be37-eb01c710837f-kube-api-access-8nmj6\") pod \"must-gather-xpbmv\" (UID: \"0b1a819f-4d2f-4ed6-be37-eb01c710837f\") " pod="openshift-must-gather-2vwnw/must-gather-xpbmv" Sep 08 01:02:56.601807 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.601713 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0b1a819f-4d2f-4ed6-be37-eb01c710837f-must-gather-output\") pod \"must-gather-xpbmv\" (UID: \"0b1a819f-4d2f-4ed6-be37-eb01c710837f\") " pod="openshift-must-gather-2vwnw/must-gather-xpbmv" Sep 08 01:02:56.702397 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.702365 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nmj6\" (UniqueName: \"kubernetes.io/projected/0b1a819f-4d2f-4ed6-be37-eb01c710837f-kube-api-access-8nmj6\") pod \"must-gather-xpbmv\" (UID: \"0b1a819f-4d2f-4ed6-be37-eb01c710837f\") " pod="openshift-must-gather-2vwnw/must-gather-xpbmv" Sep 08 01:02:56.702563 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.702405 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0b1a819f-4d2f-4ed6-be37-eb01c710837f-must-gather-output\") pod \"must-gather-xpbmv\" (UID: \"0b1a819f-4d2f-4ed6-be37-eb01c710837f\") " pod="openshift-must-gather-2vwnw/must-gather-xpbmv" Sep 08 01:02:56.702685 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.702669 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0b1a819f-4d2f-4ed6-be37-eb01c710837f-must-gather-output\") pod \"must-gather-xpbmv\" (UID: \"0b1a819f-4d2f-4ed6-be37-eb01c710837f\") " pod="openshift-must-gather-2vwnw/must-gather-xpbmv" Sep 08 01:02:56.722224 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.722195 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nmj6\" (UniqueName: \"kubernetes.io/projected/0b1a819f-4d2f-4ed6-be37-eb01c710837f-kube-api-access-8nmj6\") pod \"must-gather-xpbmv\" (UID: \"0b1a819f-4d2f-4ed6-be37-eb01c710837f\") " pod="openshift-must-gather-2vwnw/must-gather-xpbmv" Sep 08 01:02:56.881810 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:56.881726 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vwnw/must-gather-xpbmv" Sep 08 01:02:57.151760 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:57.151670 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vwnw/must-gather-xpbmv"] Sep 08 01:02:57.157645 ip-10-0-139-137 kubenswrapper[2577]: W0908 01:02:57.157615 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b1a819f_4d2f_4ed6_be37_eb01c710837f.slice/crio-b6e6e193e1ddbc2b6c304ab2bba3dd905def1821957793fae6723fb75f9f05e0 WatchSource:0}: Error finding container b6e6e193e1ddbc2b6c304ab2bba3dd905def1821957793fae6723fb75f9f05e0: Status 404 returned error can't find the container with id b6e6e193e1ddbc2b6c304ab2bba3dd905def1821957793fae6723fb75f9f05e0 Sep 08 01:02:57.159616 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:57.159601 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 01:02:57.247912 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:57.247878 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vwnw/must-gather-xpbmv" event={"ID":"0b1a819f-4d2f-4ed6-be37-eb01c710837f","Type":"ContainerStarted","Data":"b6e6e193e1ddbc2b6c304ab2bba3dd905def1821957793fae6723fb75f9f05e0"} Sep 08 01:02:58.259319 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:58.259230 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vwnw/must-gather-xpbmv" event={"ID":"0b1a819f-4d2f-4ed6-be37-eb01c710837f","Type":"ContainerStarted","Data":"eee3a957b4b982151b3946fe7aea3252c1c4e64f465c575c2c8bcc5584a9dd47"} Sep 08 01:02:58.259319 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:58.259277 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vwnw/must-gather-xpbmv" event={"ID":"0b1a819f-4d2f-4ed6-be37-eb01c710837f","Type":"ContainerStarted","Data":"8e6d801194b73915d3334d694f4a356f11c6d6375aa1b4070ae8ce67719338af"} Sep 08 01:02:58.277881 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:58.277433 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2vwnw/must-gather-xpbmv" podStartSLOduration=1.547852067 podStartE2EDuration="2.277409583s" totalImagesPullingTime="729.557516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 01:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:02:58.27584813 +0000 UTC m=+1791.971089292" watchObservedRunningTime="2026-09-08 01:02:58.277409583 +0000 UTC m=+1791.972650744" Sep 08 01:02:59.530869 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:59.530784 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sl7cg_d30a2b6e-de4c-4204-9427-08784f0330c8/global-pull-secret-syncer/0.log" Sep 08 01:02:59.625363 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:59.625334 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mxsjc_2412f205-2d19-400a-9c85-e0108f6890f5/konnectivity-agent/0.log" Sep 08 01:02:59.688373 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:02:59.688327 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-137.ec2.internal_ed147346fb2da2df0b70fefaed4bb0fc/haproxy/0.log" Sep 08 01:03:06.717461 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:06.717424 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-hbxnb_c77ca624-d279-42c8-a358-70b8ef77f29f/kube-state-metrics/0.log" Sep 08 01:03:06.740894 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:06.740861 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-hbxnb_c77ca624-d279-42c8-a358-70b8ef77f29f/kube-rbac-proxy-main/0.log" Sep 08 01:03:06.775703 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:06.775658 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-hbxnb_c77ca624-d279-42c8-a358-70b8ef77f29f/kube-rbac-proxy-self/0.log" Sep 08 01:03:06.823409 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:06.823374 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9nb9x_a2e160ae-d75d-4959-a5ab-d238b4113736/console-operator/1.log" Sep 08 01:03:06.826702 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:06.826674 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9nb9x_a2e160ae-d75d-4959-a5ab-d238b4113736/console-operator/1.log" Sep 08 01:03:07.019664 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:07.019553 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kk8cg_66e32f1b-8691-42b3-8852-835aec07fb6e/node-exporter/0.log" Sep 08 01:03:07.046569 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:07.046532 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kk8cg_66e32f1b-8691-42b3-8852-835aec07fb6e/kube-rbac-proxy/0.log" Sep 08 01:03:07.071296 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:07.071268 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kk8cg_66e32f1b-8691-42b3-8852-835aec07fb6e/init-textfile/0.log" Sep 08 01:03:09.211897 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:09.211860 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs"] Sep 08 01:03:09.215676 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:09.215647 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs" Sep 08 01:03:09.225252 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:09.225225 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs"] Sep 08 01:03:09.316632 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:09.316565 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vg8w\" (UniqueName: \"kubernetes.io/projected/44f95bae-6e37-440c-be19-c892b311124d-kube-api-access-9vg8w\") pod \"perf-node-gather-daemonset-pd9vs\" (UID: \"44f95bae-6e37-440c-be19-c892b311124d\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs" Sep 08 01:03:09.316819 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:09.316676 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/44f95bae-6e37-440c-be19-c892b311124d-proc\") pod \"perf-node-gather-daemonset-pd9vs\" (UID: \"44f95bae-6e37-440c-be19-c892b311124d\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs" Sep 08 01:03:09.316819 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:09.316720 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/44f95bae-6e37-440c-be19-c892b311124d-podres\") pod \"perf-node-gather-daemonset-pd9vs\" (UID: \"44f95bae-6e37-440c-be19-c892b311124d\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs" Sep 08 01:03:09.316819 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:09.316785 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44f95bae-6e37-440c-be19-c892b311124d-lib-modules\") pod \"perf-node-gather-daemonset-pd9vs\" (UID: \"44f95bae-6e37-440c-be19-c892b311124d\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs" Sep 08 01:03:09.316986 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:09.316879 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44f95bae-6e37-440c-be19-c892b311124d-sys\") pod \"perf-node-gather-daemonset-pd9vs\" (UID: \"44f95bae-6e37-440c-be19-c892b311124d\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs" Sep 08 01:03:09.417972 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:09.417923 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vg8w\" (UniqueName: \"kubernetes.io/projected/44f95bae-6e37-440c-be19-c892b311124d-kube-api-access-9vg8w\") pod \"perf-node-gather-daemonset-pd9vs\" (UID: \"44f95bae-6e37-440c-be19-c892b311124d\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs" Sep 08 01:03:09.418185 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:09.418032 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/44f95bae-6e37-440c-be19-c892b311124d-proc\") pod \"perf-node-gather-daemonset-pd9vs\" (UID: \"44f95bae-6e37-440c-be19-c892b311124d\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs" Sep 08 01:03:09.418185 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:09.418061 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/44f95bae-6e37-440c-be19-c892b311124d-podres\") pod \"perf-node-gather-daemonset-pd9vs\" (UID: \"44f95bae-6e37-440c-be19-c892b311124d\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs" Sep 08 01:03:09.418185 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:09.418100 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44f95bae-6e37-440c-be19-c892b311124d-lib-modules\") pod \"perf-node-gather-daemonset-pd9vs\" (UID: \"44f95bae-6e37-440c-be19-c892b311124d\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs" Sep 08 01:03:09.418185 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:09.418139 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44f95bae-6e37-440c-be19-c892b311124d-sys\") pod \"perf-node-gather-daemonset-pd9vs\" (UID: \"44f95bae-6e37-440c-be19-c892b311124d\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs" Sep 08 01:03:09.418432 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:09.418257 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/44f95bae-6e37-440c-be19-c892b311124d-sys\") pod \"perf-node-gather-daemonset-pd9vs\" (UID: \"44f95bae-6e37-440c-be19-c892b311124d\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs" Sep 08 01:03:09.418638 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:09.418613 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/44f95bae-6e37-440c-be19-c892b311124d-proc\") pod \"perf-node-gather-daemonset-pd9vs\" (UID: \"44f95bae-6e37-440c-be19-c892b311124d\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs" Sep 08 01:03:09.418761 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:09.418742 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/44f95bae-6e37-440c-be19-c892b311124d-podres\") pod \"perf-node-gather-daemonset-pd9vs\" (UID: \"44f95bae-6e37-440c-be19-c892b311124d\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs" Sep 08 01:03:09.418849 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:09.418833 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/44f95bae-6e37-440c-be19-c892b311124d-lib-modules\") pod \"perf-node-gather-daemonset-pd9vs\" (UID: \"44f95bae-6e37-440c-be19-c892b311124d\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs" Sep 08 01:03:09.427275 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:09.427241 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vg8w\" (UniqueName: \"kubernetes.io/projected/44f95bae-6e37-440c-be19-c892b311124d-kube-api-access-9vg8w\") pod \"perf-node-gather-daemonset-pd9vs\" (UID: \"44f95bae-6e37-440c-be19-c892b311124d\") " pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs" Sep 08 01:03:09.534179 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:09.534140 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs" Sep 08 01:03:09.567862 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:09.567816 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9nb9x_a2e160ae-d75d-4959-a5ab-d238b4113736/console-operator/1.log" Sep 08 01:03:09.572761 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:09.572653 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-9nb9x_a2e160ae-d75d-4959-a5ab-d238b4113736/console-operator/2.log" Sep 08 01:03:09.824175 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:09.824145 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs"] Sep 08 01:03:09.827226 ip-10-0-139-137 kubenswrapper[2577]: W0908 01:03:09.827192 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod44f95bae_6e37_440c_be19_c892b311124d.slice/crio-8da8f9c8a77fb0f6231beaf248e39f7c296e2e2f3c747373a39ebcfe6d3b7a01 WatchSource:0}: Error finding container 8da8f9c8a77fb0f6231beaf248e39f7c296e2e2f3c747373a39ebcfe6d3b7a01: Status 404 returned error can't find the container with id 8da8f9c8a77fb0f6231beaf248e39f7c296e2e2f3c747373a39ebcfe6d3b7a01 Sep 08 01:03:10.341060 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:10.341020 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs" event={"ID":"44f95bae-6e37-440c-be19-c892b311124d","Type":"ContainerStarted","Data":"b6e00cd8ea2ec0db8a16a587452e243aff06821a82ffe169d7ae6d3c646be729"} Sep 08 01:03:10.341489 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:10.341087 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs" event={"ID":"44f95bae-6e37-440c-be19-c892b311124d","Type":"ContainerStarted","Data":"8da8f9c8a77fb0f6231beaf248e39f7c296e2e2f3c747373a39ebcfe6d3b7a01"} Sep 08 01:03:10.341489 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:10.341117 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs" Sep 08 01:03:10.357334 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:10.357284 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs" podStartSLOduration=1.357260589 podStartE2EDuration="1.357260589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 01:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 01:03:10.355725739 +0000 UTC m=+1804.050966899" watchObservedRunningTime="2026-09-08 01:03:10.357260589 +0000 UTC m=+1804.052501747" Sep 08 01:03:10.827313 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:10.827270 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-8js6v_c33f752b-afab-4bad-8bc7-7e32c7817904/volume-data-source-validator/0.log" Sep 08 01:03:11.858717 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:11.858687 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kfd8x_840e6533-8a38-426a-bb9a-abc4cad33caa/dns/0.log" Sep 08 01:03:11.906302 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:11.906269 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kfd8x_840e6533-8a38-426a-bb9a-abc4cad33caa/kube-rbac-proxy/0.log" Sep 08 01:03:12.092816 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:12.092786 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zpwqj_2b13bec4-42f2-4fda-95df-9773dfd10505/dns-node-resolver/0.log" Sep 08 01:03:12.635747 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:12.635712 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-25zw7_3587d6fa-1292-4006-9971-9937f6de1058/node-ca/0.log" Sep 08 01:03:13.998605 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:13.998435 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-h2gwx_14d4c77a-4999-4d40-803a-80b4f77aad5d/discovery/0.log" Sep 08 01:03:14.057104 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:14.057076 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58c5b795c4-npd8z_fbf31006-fc5d-4957-9766-b89b65cee90a/kube-auth-proxy/0.log" Sep 08 01:03:15.144298 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:15.144270 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xvxpk_08d6201a-e98a-45e5-a580-8d3ab6ca3ef5/serve-healthcheck-canary/0.log" Sep 08 01:03:15.630636 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:15.630602 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hzcjc_20bb0c4e-2cd4-468f-88e8-979c012be938/kube-rbac-proxy/0.log" Sep 08 01:03:15.659417 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:15.659384 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hzcjc_20bb0c4e-2cd4-468f-88e8-979c012be938/exporter/0.log" Sep 08 01:03:15.682356 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:15.682327 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hzcjc_20bb0c4e-2cd4-468f-88e8-979c012be938/extractor/0.log" Sep 08 01:03:16.360397 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:16.359458 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2vwnw/perf-node-gather-daemonset-pd9vs" Sep 08 01:03:18.129609 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:18.129568 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7c7d4c4566-677bv_e8a588fa-2622-4fde-bdab-d7f5a7474767/manager/0.log" Sep 08 01:03:18.167409 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:18.167314 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-7c7d4c4566-677bv_e8a588fa-2622-4fde-bdab-d7f5a7474767/copy-manifests/0.log" Sep 08 01:03:18.944018 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:18.943506 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-h2gwx_14d4c77a-4999-4d40-803a-80b4f77aad5d/discovery/0.log" Sep 08 01:03:18.977750 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:18.977602 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58c5b795c4-npd8z_fbf31006-fc5d-4957-9766-b89b65cee90a/kube-auth-proxy/0.log" Sep 08 01:03:19.779986 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:19.779956 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-55885fcb99-2fcml_4b732ebd-9724-46b7-b315-ae24c46ef456/manager/0.log" Sep 08 01:03:20.995707 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:20.995618 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-h2gwx_14d4c77a-4999-4d40-803a-80b4f77aad5d/discovery/0.log" Sep 08 01:03:21.029356 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:21.029325 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58c5b795c4-npd8z_fbf31006-fc5d-4957-9766-b89b65cee90a/kube-auth-proxy/0.log" Sep 08 01:03:26.076463 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:26.076421 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-pwkws_2e449098-718c-4079-bc8d-cfd13c463184/kube-storage-version-migrator-operator/1.log" Sep 08 01:03:26.077383 ip-10-0-139-137 kubenswrapper[2577]: I0908 01:03:26.077361 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-pwkws_2e449098-718c-4079-bc8d-cfd13c463184/kube-storage-version-migrator-operator/0.log"