Sep 14 02:50:28.142750 ip-10-0-129-88 systemd[1]: Starting Kubernetes Kubelet... Sep 14 02:50:28.583790 ip-10-0-129-88 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 02:50:28.583790 ip-10-0-129-88 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 14 02:50:28.583790 ip-10-0-129-88 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 02:50:28.583790 ip-10-0-129-88 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 14 02:50:28.583790 ip-10-0-129-88 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 02:50:28.584815 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.584739 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 14 02:50:28.590714 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590700 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 02:50:28.590714 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590714 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 02:50:28.590779 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590718 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 02:50:28.590779 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590721 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 02:50:28.590779 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590724 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 02:50:28.590779 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590727 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 02:50:28.590779 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590730 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 02:50:28.590779 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590733 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 02:50:28.590779 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590735 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 02:50:28.590779 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590737 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 02:50:28.590779 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590741 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 02:50:28.590779 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590743 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 02:50:28.590779 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590746 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 02:50:28.590779 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590748 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 02:50:28.590779 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590751 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 02:50:28.590779 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590753 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 02:50:28.590779 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590756 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 02:50:28.590779 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590758 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 14 02:50:28.590779 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590761 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 02:50:28.590779 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590764 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 02:50:28.590779 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590767 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 02:50:28.590779 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590770 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 02:50:28.591337 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590772 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 02:50:28.591337 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590774 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 02:50:28.591337 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590777 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 02:50:28.591337 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590779 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 02:50:28.591337 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590782 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 02:50:28.591337 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590784 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 02:50:28.591337 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590787 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 02:50:28.591337 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590790 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 02:50:28.591337 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590792 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 02:50:28.591337 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590795 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 02:50:28.591337 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590798 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 02:50:28.591337 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590800 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 02:50:28.591337 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590803 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 02:50:28.591337 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590805 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 02:50:28.591337 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590808 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 02:50:28.591337 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590810 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 02:50:28.591337 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590812 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 02:50:28.591337 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590815 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 02:50:28.591337 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590818 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 02:50:28.591337 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590820 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 02:50:28.591845 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590823 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 02:50:28.591845 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590825 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 02:50:28.591845 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590827 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 02:50:28.591845 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590830 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 02:50:28.591845 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590832 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 02:50:28.591845 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590835 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 02:50:28.591845 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590837 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 02:50:28.591845 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590839 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 02:50:28.591845 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590842 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 02:50:28.591845 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590844 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 02:50:28.591845 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590847 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 02:50:28.591845 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590849 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 02:50:28.591845 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590852 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 02:50:28.591845 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590855 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 02:50:28.591845 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590858 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 02:50:28.591845 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590861 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 02:50:28.591845 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590863 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 02:50:28.591845 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590866 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 02:50:28.591845 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590869 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 02:50:28.591845 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590871 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 02:50:28.592347 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590873 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 02:50:28.592347 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590876 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 02:50:28.592347 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590878 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 02:50:28.592347 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590881 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 02:50:28.592347 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590884 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 02:50:28.592347 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590886 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 02:50:28.592347 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590889 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 02:50:28.592347 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590891 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 02:50:28.592347 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590893 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 02:50:28.592347 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590896 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 02:50:28.592347 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590901 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 02:50:28.592347 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590904 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 02:50:28.592347 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590907 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 02:50:28.592347 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590910 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 02:50:28.592347 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590913 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 02:50:28.592347 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590918 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 02:50:28.592347 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590921 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 02:50:28.592347 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590924 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 02:50:28.592347 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590927 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 02:50:28.592836 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590930 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 02:50:28.592836 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590933 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 02:50:28.592836 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590935 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 02:50:28.592836 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590938 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 02:50:28.592836 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590940 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 02:50:28.592836 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590943 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 02:50:28.592836 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590946 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 02:50:28.592836 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590949 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 02:50:28.592836 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590951 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 02:50:28.592836 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590954 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 02:50:28.592836 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590957 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 02:50:28.592836 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590961 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 02:50:28.592836 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590963 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 02:50:28.592836 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590966 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 02:50:28.592836 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590968 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 02:50:28.592836 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590971 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 02:50:28.592836 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590973 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 02:50:28.592836 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.590976 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 02:50:28.592836 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591679 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 02:50:28.592836 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591686 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 02:50:28.593363 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591689 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 02:50:28.593363 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591692 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 02:50:28.593363 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591696 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 02:50:28.593363 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591701 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 02:50:28.593363 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591704 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 02:50:28.593363 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591708 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 02:50:28.593363 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591710 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 02:50:28.593363 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591713 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 02:50:28.593363 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591716 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 14 02:50:28.593363 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591719 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 02:50:28.593363 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591721 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 02:50:28.593363 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591724 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 02:50:28.593363 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591726 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 02:50:28.593363 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591729 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 02:50:28.593363 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591732 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 02:50:28.593363 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591734 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 02:50:28.593363 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591737 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 02:50:28.593363 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591739 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 02:50:28.593363 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591744 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 02:50:28.593870 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591746 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 02:50:28.593870 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591749 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 02:50:28.593870 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591752 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 02:50:28.593870 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591754 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 02:50:28.593870 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591757 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 02:50:28.593870 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591759 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 02:50:28.593870 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591762 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 02:50:28.593870 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591764 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 02:50:28.593870 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591767 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 02:50:28.593870 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591769 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 02:50:28.593870 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591772 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 02:50:28.593870 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591774 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 02:50:28.593870 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591777 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 02:50:28.593870 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591779 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 02:50:28.593870 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591783 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 02:50:28.593870 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591786 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 02:50:28.593870 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591789 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 02:50:28.593870 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591792 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 02:50:28.593870 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591795 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 02:50:28.594333 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591800 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 02:50:28.594333 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591803 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 02:50:28.594333 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591806 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 02:50:28.594333 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591809 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 02:50:28.594333 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591811 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 02:50:28.594333 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591814 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 02:50:28.594333 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591816 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 02:50:28.594333 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591819 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 02:50:28.594333 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591821 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 02:50:28.594333 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591825 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 02:50:28.594333 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591827 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 02:50:28.594333 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591830 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 02:50:28.594333 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591833 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 02:50:28.594333 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591836 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 02:50:28.594333 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591838 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 02:50:28.594333 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591841 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 02:50:28.594333 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591843 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 02:50:28.594333 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591846 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 02:50:28.594333 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591849 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 02:50:28.594333 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591852 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 02:50:28.594832 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591854 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 02:50:28.594832 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591857 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 02:50:28.594832 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591860 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 02:50:28.594832 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591862 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 02:50:28.594832 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591865 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 02:50:28.594832 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591867 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 02:50:28.594832 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591870 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 02:50:28.594832 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591872 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 02:50:28.594832 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591875 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 02:50:28.594832 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591877 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 02:50:28.594832 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591880 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 02:50:28.594832 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591885 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 02:50:28.594832 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591888 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 02:50:28.594832 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591890 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 02:50:28.594832 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591893 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 02:50:28.594832 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591895 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 02:50:28.594832 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591897 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 02:50:28.594832 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591900 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 02:50:28.594832 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591902 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 02:50:28.594832 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591905 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 02:50:28.595317 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591907 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 02:50:28.595317 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591910 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 02:50:28.595317 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591912 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 02:50:28.595317 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591915 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 02:50:28.595317 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591918 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 02:50:28.595317 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591921 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 02:50:28.595317 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591924 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 02:50:28.595317 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591927 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 02:50:28.595317 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591929 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 02:50:28.595317 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591932 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 02:50:28.595317 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591934 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 02:50:28.595317 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591937 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 02:50:28.595317 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591939 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 02:50:28.595317 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591942 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 02:50:28.595317 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591945 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 02:50:28.595317 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591947 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 02:50:28.595317 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591951 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 02:50:28.595317 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591954 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 02:50:28.595317 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.591957 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 02:50:28.595797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592316 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 02:50:28.595797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592322 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 02:50:28.595797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592324 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 02:50:28.595797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592327 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 02:50:28.595797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592336 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 02:50:28.595797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592340 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 02:50:28.595797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592342 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 02:50:28.595797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592345 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 02:50:28.595797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592347 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 02:50:28.595797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592350 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 02:50:28.595797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592353 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 02:50:28.595797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592356 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 02:50:28.595797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592358 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 02:50:28.595797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592361 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 02:50:28.595797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592364 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 02:50:28.595797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592366 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 02:50:28.595797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592369 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 02:50:28.595797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592372 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 02:50:28.595797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592375 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 02:50:28.596275 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592378 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 02:50:28.596275 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592380 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 02:50:28.596275 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592383 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 02:50:28.596275 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592385 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 02:50:28.596275 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592388 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 02:50:28.596275 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592390 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 02:50:28.596275 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592393 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 02:50:28.596275 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592395 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 02:50:28.596275 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592398 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 02:50:28.596275 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592400 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 02:50:28.596275 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592403 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 02:50:28.596275 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592405 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 02:50:28.596275 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592408 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 02:50:28.596275 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592410 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 02:50:28.596275 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592413 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 02:50:28.596275 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592415 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 02:50:28.596275 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592418 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 02:50:28.596275 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592421 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 02:50:28.596275 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592424 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 02:50:28.596275 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592426 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 02:50:28.596790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592429 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 02:50:28.596790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592431 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 02:50:28.596790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592433 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 02:50:28.596790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592436 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 02:50:28.596790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592441 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 02:50:28.596790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592444 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 02:50:28.596790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592448 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 02:50:28.596790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592452 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 02:50:28.596790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592455 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 02:50:28.596790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592458 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 02:50:28.596790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592463 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 02:50:28.596790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592465 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 14 02:50:28.596790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592468 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 02:50:28.596790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592489 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 02:50:28.596790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592493 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 02:50:28.596790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592496 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 02:50:28.596790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592498 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 02:50:28.596790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592501 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 02:50:28.596790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592504 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 02:50:28.596790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592507 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 02:50:28.597270 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592509 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 02:50:28.597270 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592512 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 02:50:28.597270 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592514 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 02:50:28.597270 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592517 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 02:50:28.597270 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592520 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 02:50:28.597270 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592522 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 02:50:28.597270 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592525 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 02:50:28.597270 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592527 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 02:50:28.597270 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592529 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 02:50:28.597270 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592532 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 02:50:28.597270 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592534 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 02:50:28.597270 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592537 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 02:50:28.597270 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592539 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 02:50:28.597270 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592542 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 02:50:28.597270 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592544 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 02:50:28.597270 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592547 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 02:50:28.597270 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592551 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 02:50:28.597270 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592553 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 02:50:28.597270 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592556 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 02:50:28.597270 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592558 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 02:50:28.597797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592561 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 02:50:28.597797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592564 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 02:50:28.597797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592569 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 02:50:28.597797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592572 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 02:50:28.597797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592574 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 02:50:28.597797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592577 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 02:50:28.597797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592579 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 02:50:28.597797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592582 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 02:50:28.597797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592584 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 02:50:28.597797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592586 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 02:50:28.597797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592589 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 02:50:28.597797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592591 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 02:50:28.597797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592594 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 02:50:28.597797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592597 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 02:50:28.597797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592599 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 02:50:28.597797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592602 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 02:50:28.597797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592604 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 02:50:28.597797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592607 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 02:50:28.597797 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592609 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.592611 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592683 2564 flags.go:64] FLAG: --address="0.0.0.0" Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592716 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592722 2564 flags.go:64] FLAG: --anonymous-auth="true" Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592726 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592730 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592733 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592737 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592741 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592745 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592748 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592751 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592754 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592757 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592760 2564 flags.go:64] FLAG: --cgroup-root="" Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592764 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592767 2564 flags.go:64] FLAG: --client-ca-file="" Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592769 2564 flags.go:64] FLAG: --cloud-provider="external" Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592772 2564 flags.go:64] FLAG: --cluster-dns="[]" Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592776 2564 flags.go:64] FLAG: --cluster-domain="" Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592779 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592782 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592785 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 14 02:50:28.598306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592788 2564 flags.go:64] FLAG: --container-log-max-files="5" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592792 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592794 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592797 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592800 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592803 2564 flags.go:64] FLAG: --contention-profiling="false" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592806 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592809 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592815 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592818 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592822 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592825 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592827 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592830 2564 flags.go:64] FLAG: --enable-load-reader="false" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592833 2564 flags.go:64] FLAG: --enable-server="true" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592836 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592840 2564 flags.go:64] FLAG: --event-burst="100" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592843 2564 flags.go:64] FLAG: --event-qps="50" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592847 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592851 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592854 2564 flags.go:64] FLAG: --eviction-hard="" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592857 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592860 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592863 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592867 2564 flags.go:64] FLAG: --eviction-soft="" Sep 14 02:50:28.598910 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592870 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592873 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592876 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592878 2564 flags.go:64] FLAG: --experimental-mounter-path="" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592881 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592884 2564 flags.go:64] FLAG: --fail-swap-on="true" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592887 2564 flags.go:64] FLAG: --feature-gates="" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592892 2564 flags.go:64] FLAG: --file-check-frequency="20s" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592895 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592898 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592901 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592904 2564 flags.go:64] FLAG: --healthz-port="10248" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592907 2564 flags.go:64] FLAG: --help="false" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592909 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-129-88.ec2.internal" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592912 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592915 2564 flags.go:64] FLAG: --http-check-frequency="20s" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592918 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592921 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592924 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592927 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592930 2564 flags.go:64] FLAG: --image-service-endpoint="" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592932 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592935 2564 flags.go:64] FLAG: --kube-api-burst="100" Sep 14 02:50:28.599529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592939 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592942 2564 flags.go:64] FLAG: --kube-api-qps="50" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592946 2564 flags.go:64] FLAG: --kube-reserved="" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592949 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592952 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592955 2564 flags.go:64] FLAG: --kubelet-cgroups="" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592957 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592960 2564 flags.go:64] FLAG: --lock-file="" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592965 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592968 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592970 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592976 2564 flags.go:64] FLAG: --log-json-split-stream="false" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592979 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592981 2564 flags.go:64] FLAG: --log-text-split-stream="false" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592984 2564 flags.go:64] FLAG: --logging-format="text" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592987 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592990 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592993 2564 flags.go:64] FLAG: --manifest-url="" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.592996 2564 flags.go:64] FLAG: --manifest-url-header="" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593000 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593003 2564 flags.go:64] FLAG: --max-open-files="1000000" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593006 2564 flags.go:64] FLAG: --max-pods="110" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593009 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593012 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593015 2564 flags.go:64] FLAG: --memory-manager-policy="None" Sep 14 02:50:28.600143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593018 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593020 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593023 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593026 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593033 2564 flags.go:64] FLAG: --node-status-max-images="50" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593036 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593039 2564 flags.go:64] FLAG: --oom-score-adj="-999" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593042 2564 flags.go:64] FLAG: --pod-cidr="" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593045 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593050 2564 flags.go:64] FLAG: --pod-manifest-path="" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593054 2564 flags.go:64] FLAG: --pod-max-pids="-1" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593058 2564 flags.go:64] FLAG: --pods-per-core="0" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593060 2564 flags.go:64] FLAG: --port="10250" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593063 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593066 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-099e5b0e568d2252a" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593070 2564 flags.go:64] FLAG: --qos-reserved="" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593073 2564 flags.go:64] FLAG: --read-only-port="10255" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593076 2564 flags.go:64] FLAG: --register-node="true" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593079 2564 flags.go:64] FLAG: --register-with-taints="" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593083 2564 flags.go:64] FLAG: --registry-burst="10" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593085 2564 flags.go:64] FLAG: --registry-qps="5" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593088 2564 flags.go:64] FLAG: --reserved-cpus="" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593091 2564 flags.go:64] FLAG: --reserved-memory="" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593094 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593097 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 14 02:50:28.600797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593100 2564 flags.go:64] FLAG: --rotate-certificates="false" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593103 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593106 2564 flags.go:64] FLAG: --runonce="false" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593108 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593111 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593114 2564 flags.go:64] FLAG: --seccomp-default="false" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593116 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593119 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593122 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593125 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593128 2564 flags.go:64] FLAG: --storage-driver-password="root" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593131 2564 flags.go:64] FLAG: --storage-driver-secure="false" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593134 2564 flags.go:64] FLAG: --storage-driver-table="stats" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593136 2564 flags.go:64] FLAG: --storage-driver-user="root" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593139 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593142 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593145 2564 flags.go:64] FLAG: --system-cgroups="" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593149 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593155 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593157 2564 flags.go:64] FLAG: --tls-cert-file="" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593160 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593164 2564 flags.go:64] FLAG: --tls-min-version="" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593169 2564 flags.go:64] FLAG: --tls-private-key-file="" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593172 2564 flags.go:64] FLAG: --topology-manager-policy="none" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593174 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 14 02:50:28.601407 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593177 2564 flags.go:64] FLAG: --topology-manager-scope="container" Sep 14 02:50:28.602080 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593180 2564 flags.go:64] FLAG: --v="2" Sep 14 02:50:28.602080 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593184 2564 flags.go:64] FLAG: --version="false" Sep 14 02:50:28.602080 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593188 2564 flags.go:64] FLAG: --vmodule="" Sep 14 02:50:28.602080 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593191 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 14 02:50:28.602080 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593194 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 14 02:50:28.602080 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593307 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 02:50:28.602080 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593310 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 02:50:28.602080 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593313 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 02:50:28.602080 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593316 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 02:50:28.602080 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593319 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 02:50:28.602080 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593321 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 02:50:28.602080 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593324 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 02:50:28.602080 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593326 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 02:50:28.602080 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593329 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 02:50:28.602080 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593331 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 02:50:28.602080 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593334 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 02:50:28.602080 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593337 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 02:50:28.602080 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593339 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 02:50:28.602080 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593342 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 02:50:28.602080 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593344 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 02:50:28.602080 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593347 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 02:50:28.602610 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593349 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 02:50:28.602610 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593352 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 02:50:28.602610 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593357 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 02:50:28.602610 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593359 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 02:50:28.602610 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593362 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 02:50:28.602610 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593365 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 02:50:28.602610 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593367 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 02:50:28.602610 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593371 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 02:50:28.602610 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593374 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 02:50:28.602610 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593377 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 02:50:28.602610 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593380 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 02:50:28.602610 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593383 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 02:50:28.602610 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593386 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 02:50:28.602610 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593388 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 02:50:28.602610 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593390 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 02:50:28.602610 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593393 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 02:50:28.602610 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593396 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 02:50:28.602610 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593398 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 02:50:28.602610 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593401 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 02:50:28.602610 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593403 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 02:50:28.603105 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593407 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 02:50:28.603105 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593411 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 14 02:50:28.603105 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593414 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 02:50:28.603105 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593417 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 02:50:28.603105 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593420 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 02:50:28.603105 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593422 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 02:50:28.603105 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593425 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 02:50:28.603105 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593428 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 02:50:28.603105 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593430 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 02:50:28.603105 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593433 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 02:50:28.603105 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593435 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 02:50:28.603105 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593438 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 02:50:28.603105 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593440 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 02:50:28.603105 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593442 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 02:50:28.603105 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593446 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 02:50:28.603105 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593449 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 02:50:28.603105 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593451 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 02:50:28.603105 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593454 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 02:50:28.603105 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593456 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 02:50:28.603597 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593459 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 02:50:28.603597 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593462 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 02:50:28.603597 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593465 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 02:50:28.603597 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593467 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 02:50:28.603597 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593470 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 02:50:28.603597 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593486 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 02:50:28.603597 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593489 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 02:50:28.603597 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593492 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 02:50:28.603597 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593494 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 02:50:28.603597 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593497 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 02:50:28.603597 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593499 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 02:50:28.603597 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593502 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 02:50:28.603597 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593504 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 02:50:28.603597 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593507 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 02:50:28.603597 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593511 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 02:50:28.603597 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593514 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 02:50:28.603597 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593517 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 02:50:28.603597 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593520 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 02:50:28.603597 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593523 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 02:50:28.604068 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593526 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 02:50:28.604068 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593528 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 02:50:28.604068 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593531 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 02:50:28.604068 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593534 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 02:50:28.604068 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593536 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 02:50:28.604068 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593540 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 02:50:28.604068 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593543 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 02:50:28.604068 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593546 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 02:50:28.604068 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593549 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 02:50:28.604068 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593552 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 02:50:28.604068 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593555 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 02:50:28.604068 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593558 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 02:50:28.604068 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593560 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 02:50:28.604068 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593563 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 02:50:28.604068 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593566 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 02:50:28.604068 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593568 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 02:50:28.604068 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593571 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 02:50:28.604068 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593573 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 02:50:28.604068 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593576 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 02:50:28.604567 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593578 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 02:50:28.604567 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593581 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 02:50:28.604567 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593583 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 02:50:28.604567 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593586 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 02:50:28.604567 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593588 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 02:50:28.604567 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.593591 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 02:50:28.604567 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.593596 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 02:50:28.604567 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.601108 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 14 02:50:28.604567 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.601122 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 14 02:50:28.604567 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.601992 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 14 02:50:28.604567 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.602000 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 14 02:50:28.604567 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602055 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 02:50:28.604567 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602060 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 02:50:28.604567 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602063 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 02:50:28.604913 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602066 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 02:50:28.604913 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602069 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 02:50:28.604913 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602072 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 02:50:28.604913 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602075 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 02:50:28.604913 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602078 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 02:50:28.604913 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602080 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 02:50:28.604913 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602083 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 02:50:28.604913 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602086 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 14 02:50:28.604913 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602088 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 02:50:28.604913 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602091 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 02:50:28.604913 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602094 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 02:50:28.604913 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602096 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 02:50:28.604913 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602099 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 02:50:28.604913 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602101 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 02:50:28.604913 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602104 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 02:50:28.604913 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602106 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 02:50:28.604913 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602109 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 02:50:28.604913 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602111 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 02:50:28.604913 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602114 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 02:50:28.604913 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602116 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 02:50:28.605432 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602119 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 02:50:28.605432 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602121 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 02:50:28.605432 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602124 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 02:50:28.605432 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602126 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 02:50:28.605432 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602129 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 02:50:28.605432 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602133 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 02:50:28.605432 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602138 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 02:50:28.605432 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602141 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 02:50:28.605432 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602144 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 02:50:28.605432 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602147 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 02:50:28.605432 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602150 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 02:50:28.605432 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602153 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 02:50:28.605432 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602155 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 02:50:28.605432 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602158 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 02:50:28.605432 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602161 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 02:50:28.605432 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602163 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 02:50:28.605432 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602166 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 02:50:28.605432 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602169 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 02:50:28.605432 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602172 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 02:50:28.606030 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602174 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 02:50:28.606030 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602177 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 02:50:28.606030 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602180 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 02:50:28.606030 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602183 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 02:50:28.606030 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602188 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 02:50:28.606030 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602191 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 02:50:28.606030 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602194 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 02:50:28.606030 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602197 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 02:50:28.606030 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602200 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 02:50:28.606030 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602202 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 02:50:28.606030 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602205 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 02:50:28.606030 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602207 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 02:50:28.606030 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602210 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 02:50:28.606030 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602212 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 02:50:28.606030 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602215 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 02:50:28.606030 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602218 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 02:50:28.606030 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602220 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 02:50:28.606030 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602223 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 02:50:28.606030 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602225 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 02:50:28.606503 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602228 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 02:50:28.606503 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602231 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 02:50:28.606503 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602234 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 02:50:28.606503 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602236 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 02:50:28.606503 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602239 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 02:50:28.606503 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602241 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 02:50:28.606503 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602244 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 02:50:28.606503 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602246 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 02:50:28.606503 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602248 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 02:50:28.606503 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602251 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 02:50:28.606503 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602253 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 02:50:28.606503 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602256 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 02:50:28.606503 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602259 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 02:50:28.606503 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602261 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 02:50:28.606503 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602263 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 02:50:28.606503 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602265 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 02:50:28.606503 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602268 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 02:50:28.606503 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602270 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 02:50:28.606503 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602273 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 02:50:28.606503 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602275 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 02:50:28.606993 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602278 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 02:50:28.606993 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602280 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 02:50:28.606993 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602283 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 02:50:28.606993 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602285 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 02:50:28.606993 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602288 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 02:50:28.606993 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602290 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 02:50:28.606993 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602293 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 02:50:28.606993 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602295 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 02:50:28.606993 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602298 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 02:50:28.606993 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602300 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 02:50:28.606993 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602303 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 02:50:28.606993 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602305 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 02:50:28.606993 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602309 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 02:50:28.606993 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602311 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 02:50:28.606993 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602316 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 02:50:28.606993 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602318 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 02:50:28.606993 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602321 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 02:50:28.606993 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602323 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 02:50:28.607426 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.602329 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 02:50:28.607426 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602438 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 02:50:28.607426 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602443 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 02:50:28.607426 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602447 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 02:50:28.607426 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602450 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 02:50:28.607426 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602453 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 02:50:28.607426 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602456 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 02:50:28.607426 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602459 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 02:50:28.607426 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602461 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 02:50:28.607426 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602464 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 02:50:28.607426 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602467 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 02:50:28.607426 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602469 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 02:50:28.607426 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602484 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 02:50:28.607426 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602487 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 02:50:28.607790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602489 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 02:50:28.607790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602492 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 02:50:28.607790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602495 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 02:50:28.607790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602497 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 02:50:28.607790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602500 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 02:50:28.607790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602502 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 02:50:28.607790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602505 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 02:50:28.607790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602508 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 02:50:28.607790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602510 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 02:50:28.607790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602513 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 02:50:28.607790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602515 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 02:50:28.607790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602518 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 02:50:28.607790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602521 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 02:50:28.607790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602523 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 02:50:28.607790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602526 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 02:50:28.607790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602529 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 02:50:28.607790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602532 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 02:50:28.607790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602534 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 02:50:28.607790 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602537 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 02:50:28.608249 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602539 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 02:50:28.608249 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602542 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 02:50:28.608249 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602544 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 02:50:28.608249 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602547 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 02:50:28.608249 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602549 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 02:50:28.608249 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602552 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 02:50:28.608249 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602554 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 02:50:28.608249 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602557 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 02:50:28.608249 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602559 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 02:50:28.608249 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602562 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 02:50:28.608249 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602564 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 02:50:28.608249 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602567 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 02:50:28.608249 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602569 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 02:50:28.608249 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602573 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 02:50:28.608249 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602577 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 02:50:28.608249 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602579 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 02:50:28.608249 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602582 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 02:50:28.608249 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602585 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 02:50:28.608249 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602588 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 02:50:28.608728 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602590 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 02:50:28.608728 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602593 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 02:50:28.608728 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602595 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 02:50:28.608728 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602598 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 02:50:28.608728 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602601 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 02:50:28.608728 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602603 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 02:50:28.608728 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602607 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 02:50:28.608728 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602609 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 02:50:28.608728 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602612 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 02:50:28.608728 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602615 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 02:50:28.608728 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602618 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 02:50:28.608728 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602621 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 02:50:28.608728 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602623 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 14 02:50:28.608728 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602626 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 02:50:28.608728 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602629 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 02:50:28.608728 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602631 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 02:50:28.608728 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602633 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 02:50:28.608728 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602636 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 02:50:28.608728 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602638 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 02:50:28.608728 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602640 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 02:50:28.609240 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602643 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 02:50:28.609240 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602645 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 02:50:28.609240 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602648 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 02:50:28.609240 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602650 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 02:50:28.609240 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602653 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 02:50:28.609240 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602656 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 02:50:28.609240 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602659 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 02:50:28.609240 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602661 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 02:50:28.609240 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602664 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 02:50:28.609240 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602666 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 02:50:28.609240 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602669 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 02:50:28.609240 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602671 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 02:50:28.609240 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602673 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 02:50:28.609240 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602676 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 02:50:28.609240 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602679 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 02:50:28.609240 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602681 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 02:50:28.609240 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602684 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 02:50:28.609240 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602686 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 02:50:28.609240 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602689 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 02:50:28.609240 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602691 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 02:50:28.609240 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602694 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 02:50:28.609764 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602696 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 02:50:28.609764 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602700 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 02:50:28.609764 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602704 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 02:50:28.609764 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602707 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 02:50:28.609764 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602709 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 02:50:28.609764 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602712 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 02:50:28.609764 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:28.602714 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 02:50:28.609764 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.602719 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 02:50:28.609764 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.602829 2564 server.go:962] "Client rotation is on, will bootstrap in background" Sep 14 02:50:28.609764 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.605145 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 14 02:50:28.609764 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.606159 2564 server.go:1019] "Starting client certificate rotation" Sep 14 02:50:28.609764 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.606256 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 14 02:50:28.609764 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.607132 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 14 02:50:28.610084 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.609711 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 14 02:50:28.615856 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.615829 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 14 02:50:28.629501 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.629467 2564 log.go:25] "Validated CRI v1 runtime API" Sep 14 02:50:28.632742 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.632729 2564 log.go:25] "Validated CRI v1 image API" Sep 14 02:50:28.633538 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.633523 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 14 02:50:28.637358 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.637336 2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7a78bbd1-11fe-41df-be47-efdf1b0d4e49:/dev/nvme0n1p3 bcbd6965-e309-426b-bb3b-e7db181fbdd6:/dev/nvme0n1p4] Sep 14 02:50:28.637420 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.637358 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 14 02:50:28.643043 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.642936 2564 manager.go:216] Machine: {Timestamp:2026-09-14 02:50:28.641629636 +0000 UTC m=+0.371288749 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3198227 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21c67daec4b629522aebca3dc9d0b5 SystemUUID:ec21c67d-aec4-b629-522a-ebca3dc9d0b5 BootID:d4e0c1a3-8664-4c75-bb5b-63ce5bdea56a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fe:ee:b3:17 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fe:ee:b3:17 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:58:e1:be:cf:a3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 14 02:50:28.643043 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.643042 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 14 02:50:28.643145 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.643138 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 14 02:50:28.646505 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.646485 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 14 02:50:28.646643 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.646508 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-88.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 14 02:50:28.647349 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.647339 2564 topology_manager.go:138] "Creating topology manager with none policy" Sep 14 02:50:28.647379 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.647351 2564 container_manager_linux.go:309] "Creating device plugin manager" Sep 14 02:50:28.647379 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.647359 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 14 02:50:28.647379 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.647372 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 14 02:50:28.648656 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.648645 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 14 02:50:28.654155 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.654144 2564 state_mem.go:36] "Initialized new in-memory state store" Sep 14 02:50:28.654270 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.654261 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 14 02:50:28.654310 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.654290 2564 kubelet.go:486] "Attempting to sync node with API server" Sep 14 02:50:28.654310 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.654301 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 14 02:50:28.654364 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.654311 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 14 02:50:28.654364 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.654320 2564 kubelet.go:398] "Adding apiserver pod source" Sep 14 02:50:28.654364 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.654327 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 14 02:50:28.654660 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.654646 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 14 02:50:28.654701 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.654675 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 14 02:50:28.657606 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.657592 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 14 02:50:28.657889 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.657874 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 14 02:50:28.660424 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.660409 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 14 02:50:28.660497 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.660451 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 14 02:50:28.662533 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.662520 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 14 02:50:28.662569 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.662544 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 14 02:50:28.662569 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.662553 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 14 02:50:28.662569 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.662562 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 14 02:50:28.662651 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.662574 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 14 02:50:28.662651 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.662579 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 14 02:50:28.662651 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.662592 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 14 02:50:28.662651 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.662597 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 14 02:50:28.662651 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.662605 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 14 02:50:28.662651 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.662611 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 14 02:50:28.662651 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.662627 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 14 02:50:28.662651 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.662632 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 14 02:50:28.663517 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.663507 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 14 02:50:28.663553 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.663518 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 14 02:50:28.667094 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.667074 2564 server.go:1268] "Started kubelet" Sep 14 02:50:28.667192 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.667132 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 14 02:50:28.667349 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.667318 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 14 02:50:28.667414 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.667374 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 14 02:50:28.667843 ip-10-0-129-88 systemd[1]: Started Kubernetes Kubelet. Sep 14 02:50:28.668936 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.668916 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 14 02:50:28.670842 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.670795 2564 server.go:310] "Adding debug handlers to kubelet server" Sep 14 02:50:28.671635 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.671619 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-88.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 14 02:50:28.672181 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:28.672154 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-88.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 14 02:50:28.672286 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:28.672159 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 14 02:50:28.672286 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:28.672157 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 14 02:50:28.676180 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.676162 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 14 02:50:28.676838 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.676819 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 14 02:50:28.676942 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:28.676922 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-88.ec2.internal\" not found" Sep 14 02:50:28.677241 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:28.676067 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-88.ec2.internal.18d51030c90ae434 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-88.ec2.internal,UID:ip-10-0-129-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-88.ec2.internal,},FirstTimestamp:2026-09-14 02:50:28.667049012 +0000 UTC m=+0.396708120,LastTimestamp:2026-09-14 02:50:28.667049012 +0000 UTC m=+0.396708120,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-88.ec2.internal,}" Sep 14 02:50:28.677593 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.677575 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 14 02:50:28.677593 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.677579 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 14 02:50:28.677730 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.677602 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 14 02:50:28.677730 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.677650 2564 factory.go:55] Registering systemd factory Sep 14 02:50:28.677730 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.677666 2564 factory.go:223] Registration of the systemd container factory successfully Sep 14 02:50:28.677877 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.677861 2564 reconstruct.go:97] "Volume reconstruction finished" Sep 14 02:50:28.677915 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.677878 2564 reconciler.go:29] "Reconciler: start to sync state" Sep 14 02:50:28.677915 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.677905 2564 factory.go:153] Registering CRI-O factory Sep 14 02:50:28.677915 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.677916 2564 factory.go:223] Registration of the crio container factory successfully Sep 14 02:50:28.677997 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.677957 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 14 02:50:28.677997 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.677971 2564 factory.go:103] Registering Raw factory Sep 14 02:50:28.677997 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.677980 2564 manager.go:1196] Started watching for new ooms in manager Sep 14 02:50:28.679544 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.679528 2564 manager.go:349] Starting recovery of all containers Sep 14 02:50:28.689383 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.689368 2564 manager.go:354] Recovery completed Sep 14 02:50:28.692806 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:28.692786 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-88.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 14 02:50:28.692879 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:28.692813 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 14 02:50:28.693236 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.693225 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 02:50:28.694780 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.694754 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasSufficientMemory" Sep 14 02:50:28.694853 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.694794 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 02:50:28.694853 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.694808 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasSufficientPID" Sep 14 02:50:28.695331 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.695315 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 14 02:50:28.695331 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.695329 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 14 02:50:28.695331 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.695332 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 02:50:28.695470 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.695343 2564 state_mem.go:36] "Initialized new in-memory state store" Sep 14 02:50:28.696069 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.696054 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasSufficientMemory" Sep 14 02:50:28.696140 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.696085 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 02:50:28.696140 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.696101 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasSufficientPID" Sep 14 02:50:28.697529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.697516 2564 policy_none.go:49] "None policy: Start" Sep 14 02:50:28.697585 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.697532 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 14 02:50:28.697585 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.697548 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 14 02:50:28.699532 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.699520 2564 policy_none.go:47] "Start" Sep 14 02:50:28.706145 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:28.706077 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-88.ec2.internal.18d51030cab20375 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-88.ec2.internal,UID:ip-10-0-129-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-88.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-88.ec2.internal,},FirstTimestamp:2026-09-14 02:50:28.694778741 +0000 UTC m=+0.424437849,LastTimestamp:2026-09-14 02:50:28.694778741 +0000 UTC m=+0.424437849,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-88.ec2.internal,}" Sep 14 02:50:28.717890 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:28.717818 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-88.ec2.internal.18d51030cab25d22 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-88.ec2.internal,UID:ip-10-0-129-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-88.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-88.ec2.internal,},FirstTimestamp:2026-09-14 02:50:28.694801698 +0000 UTC m=+0.424460805,LastTimestamp:2026-09-14 02:50:28.694801698 +0000 UTC m=+0.424460805,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-88.ec2.internal,}" Sep 14 02:50:28.755147 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.735992 2564 manager.go:337] "Starting Device Plugin manager" Sep 14 02:50:28.755147 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:28.736025 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 14 02:50:28.755147 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.736037 2564 server.go:87] "Starting device plugin registration server" Sep 14 02:50:28.755147 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.736276 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 14 02:50:28.755147 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.736290 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 14 02:50:28.755147 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.736374 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 14 02:50:28.755147 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.736471 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 14 02:50:28.755147 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.736497 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 14 02:50:28.755147 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:28.736805 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-88.ec2.internal.18d51030cab28767 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-88.ec2.internal,UID:ip-10-0-129-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-88.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-88.ec2.internal,},FirstTimestamp:2026-09-14 02:50:28.694812519 +0000 UTC m=+0.424471626,LastTimestamp:2026-09-14 02:50:28.694812519 +0000 UTC m=+0.424471626,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-88.ec2.internal,}" Sep 14 02:50:28.755147 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:28.737098 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 14 02:50:28.755147 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:28.737164 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-88.ec2.internal\" not found" Sep 14 02:50:28.755147 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.741946 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 14 02:50:28.755147 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.743233 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 14 02:50:28.755147 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.743273 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 14 02:50:28.755147 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.743291 2564 kubelet.go:2448] "Starting kubelet main sync loop" Sep 14 02:50:28.755147 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:28.743325 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 14 02:50:28.755147 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:28.747280 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 14 02:50:28.755147 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:28.747400 2564 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-88.ec2.internal.18d51030cab20375\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-88.ec2.internal.18d51030cab20375 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-88.ec2.internal,UID:ip-10-0-129-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-88.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-88.ec2.internal,},FirstTimestamp:2026-09-14 02:50:28.694778741 +0000 UTC m=+0.424437849,LastTimestamp:2026-09-14 02:50:28.696069407 +0000 UTC m=+0.425728516,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-88.ec2.internal,}" Sep 14 02:50:28.755147 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:28.754394 2564 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-88.ec2.internal.18d51030cab25d22\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-88.ec2.internal.18d51030cab25d22 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-88.ec2.internal,UID:ip-10-0-129-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-88.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-88.ec2.internal,},FirstTimestamp:2026-09-14 02:50:28.694801698 +0000 UTC m=+0.424460805,LastTimestamp:2026-09-14 02:50:28.696091555 +0000 UTC m=+0.425750664,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-88.ec2.internal,}" Sep 14 02:50:28.760683 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:28.760629 2564 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-88.ec2.internal.18d51030cab28767\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-88.ec2.internal.18d51030cab28767 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-88.ec2.internal,UID:ip-10-0-129-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-88.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-88.ec2.internal,},FirstTimestamp:2026-09-14 02:50:28.694812519 +0000 UTC m=+0.424471626,LastTimestamp:2026-09-14 02:50:28.696107149 +0000 UTC m=+0.425766259,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-88.ec2.internal,}" Sep 14 02:50:28.772421 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:28.772366 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-88.ec2.internal.18d51030cd5033dd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-88.ec2.internal,UID:ip-10-0-129-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-129-88.ec2.internal,},FirstTimestamp:2026-09-14 02:50:28.738700253 +0000 UTC m=+0.468359348,LastTimestamp:2026-09-14 02:50:28.738700253 +0000 UTC m=+0.468359348,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-88.ec2.internal,}" Sep 14 02:50:28.789073 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.789053 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5j94q" Sep 14 02:50:28.808815 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.808795 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5j94q" Sep 14 02:50:28.836533 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.836493 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 02:50:28.837303 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.837290 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasSufficientMemory" Sep 14 02:50:28.837340 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.837315 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 02:50:28.837340 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.837331 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasSufficientPID" Sep 14 02:50:28.837434 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.837357 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-88.ec2.internal" Sep 14 02:50:28.843388 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.843366 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-88.ec2.internal"] Sep 14 02:50:28.843436 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.843426 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 02:50:28.844859 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.844843 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasSufficientMemory" Sep 14 02:50:28.844933 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.844872 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 02:50:28.844933 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.844885 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasSufficientPID" Sep 14 02:50:28.847195 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.847183 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 02:50:28.847320 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.847304 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.ec2.internal" Sep 14 02:50:28.847375 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.847340 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 02:50:28.847899 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.847881 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasSufficientMemory" Sep 14 02:50:28.847979 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.847907 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 02:50:28.847979 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.847918 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasSufficientPID" Sep 14 02:50:28.847979 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.847885 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasSufficientMemory" Sep 14 02:50:28.847979 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.847979 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 02:50:28.848097 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.847989 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasSufficientPID" Sep 14 02:50:28.850136 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.850120 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.ec2.internal" Sep 14 02:50:28.850202 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.850160 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 02:50:28.850838 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.850807 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasSufficientMemory" Sep 14 02:50:28.850838 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.850835 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 02:50:28.850968 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.850850 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasSufficientPID" Sep 14 02:50:28.861469 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.861452 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-88.ec2.internal" Sep 14 02:50:28.861469 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:28.861487 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-88.ec2.internal\": node \"ip-10-0-129-88.ec2.internal\" not found" Sep 14 02:50:28.883026 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:28.883002 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-88.ec2.internal\" not found" node="ip-10-0-129-88.ec2.internal" Sep 14 02:50:28.887253 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:28.887237 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-88.ec2.internal\" not found" node="ip-10-0-129-88.ec2.internal" Sep 14 02:50:28.979547 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.979526 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52af06832f4006a522bf3f805dd3bd2d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-88.ec2.internal\" (UID: \"52af06832f4006a522bf3f805dd3bd2d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.ec2.internal" Sep 14 02:50:28.979633 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.979550 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52af06832f4006a522bf3f805dd3bd2d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-88.ec2.internal\" (UID: \"52af06832f4006a522bf3f805dd3bd2d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.ec2.internal" Sep 14 02:50:28.979633 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:28.979572 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/edcf5b83cf757b447b27e9d1ae1ebd2b-config\") pod \"kube-apiserver-proxy-ip-10-0-129-88.ec2.internal\" (UID: \"edcf5b83cf757b447b27e9d1ae1ebd2b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.ec2.internal" Sep 14 02:50:29.080401 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:29.080378 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52af06832f4006a522bf3f805dd3bd2d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-88.ec2.internal\" (UID: \"52af06832f4006a522bf3f805dd3bd2d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.ec2.internal" Sep 14 02:50:29.080503 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:29.080402 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52af06832f4006a522bf3f805dd3bd2d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-88.ec2.internal\" (UID: \"52af06832f4006a522bf3f805dd3bd2d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.ec2.internal" Sep 14 02:50:29.080503 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:29.080419 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/edcf5b83cf757b447b27e9d1ae1ebd2b-config\") pod \"kube-apiserver-proxy-ip-10-0-129-88.ec2.internal\" (UID: \"edcf5b83cf757b447b27e9d1ae1ebd2b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.ec2.internal" Sep 14 02:50:29.080503 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:29.080454 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/edcf5b83cf757b447b27e9d1ae1ebd2b-config\") pod \"kube-apiserver-proxy-ip-10-0-129-88.ec2.internal\" (UID: \"edcf5b83cf757b447b27e9d1ae1ebd2b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.ec2.internal" Sep 14 02:50:29.080626 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:29.080506 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52af06832f4006a522bf3f805dd3bd2d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-88.ec2.internal\" (UID: \"52af06832f4006a522bf3f805dd3bd2d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.ec2.internal" Sep 14 02:50:29.080626 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:29.080512 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/52af06832f4006a522bf3f805dd3bd2d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-88.ec2.internal\" (UID: \"52af06832f4006a522bf3f805dd3bd2d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.ec2.internal" Sep 14 02:50:29.123692 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:29.123639 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-88.ec2.internal\" not found" Sep 14 02:50:29.186206 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:29.186182 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.ec2.internal" Sep 14 02:50:29.191383 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:29.191367 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.ec2.internal" Sep 14 02:50:29.224645 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:29.224005 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-88.ec2.internal\" not found" Sep 14 02:50:29.324540 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:29.324506 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-88.ec2.internal\" not found" Sep 14 02:50:29.424947 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:29.424899 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-88.ec2.internal\" not found" Sep 14 02:50:29.525351 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:29.525326 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-88.ec2.internal\" not found" Sep 14 02:50:29.605854 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:29.605833 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 14 02:50:29.626064 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:29.626044 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-88.ec2.internal\" not found" Sep 14 02:50:29.689727 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:29.689686 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 02:50:29.725339 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:29.725323 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 02:50:29.726261 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:29.726247 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-88.ec2.internal\" not found" Sep 14 02:50:29.726512 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:29.726497 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasSufficientMemory" Sep 14 02:50:29.726572 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:29.726531 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 02:50:29.726572 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:29.726544 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeHasSufficientPID" Sep 14 02:50:29.812789 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:29.812750 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-13 02:45:28 +0000 UTC" deadline="2028-04-21 06:00:49.71893172 +0000 UTC" Sep 14 02:50:29.812789 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:29.812778 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14043h10m19.906156712s" Sep 14 02:50:29.826915 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:29.826887 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-88.ec2.internal\" not found" Sep 14 02:50:29.843895 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:29.843880 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 02:50:29.850086 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:29.850060 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedcf5b83cf757b447b27e9d1ae1ebd2b.slice/crio-b757eb37e7fbd2219007aa6a6dda002f6c56fc511631a1d6526476862eb92201 WatchSource:0}: Error finding container b757eb37e7fbd2219007aa6a6dda002f6c56fc511631a1d6526476862eb92201: Status 404 returned error can't find the container with id b757eb37e7fbd2219007aa6a6dda002f6c56fc511631a1d6526476862eb92201 Sep 14 02:50:29.854195 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:29.854182 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 02:50:29.927839 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:29.927807 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-88.ec2.internal\" not found" Sep 14 02:50:30.028321 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:30.028247 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-88.ec2.internal\" not found" Sep 14 02:50:30.129421 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:30.129392 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-88.ec2.internal\" not found" Sep 14 02:50:30.202615 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.202589 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 02:50:30.230506 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:30.230470 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-88.ec2.internal\" not found" Sep 14 02:50:30.265675 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.265649 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 02:50:30.277773 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.277750 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.ec2.internal" Sep 14 02:50:30.296937 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.296878 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 14 02:50:30.300300 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.300285 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.ec2.internal" Sep 14 02:50:30.319822 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.319804 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 14 02:50:30.655683 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.655602 2564 apiserver.go:52] "Watching apiserver" Sep 14 02:50:30.665416 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.665390 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 14 02:50:30.668327 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.668287 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-jrk97","openshift-cluster-node-tuning-operator/tuned-67g56","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.ec2.internal","openshift-multus/multus-additional-cni-plugins-fjn2n","openshift-network-diagnostics/network-check-target-5hcgk","openshift-network-operator/iptables-alerter-rxxxt","openshift-ovn-kubernetes/ovnkube-node-5bwv6","kube-system/global-pull-secret-syncer-mq8rq","kube-system/konnectivity-agent-6sn84","kube-system/kube-apiserver-proxy-ip-10-0-129-88.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx","openshift-dns/node-resolver-xgj42","openshift-image-registry/node-ca-tjwgv","openshift-multus/multus-n5mdt"] Sep 14 02:50:30.671712 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.671681 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.674577 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.674552 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tjwgv" Sep 14 02:50:30.675879 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.675846 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vxp5l\"" Sep 14 02:50:30.676117 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.676056 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 14 02:50:30.676310 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.676291 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 14 02:50:30.676531 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.676513 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 14 02:50:30.676821 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.676802 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 14 02:50:30.677012 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.676995 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 14 02:50:30.677235 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.677214 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 14 02:50:30.679489 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.677998 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.679489 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.678860 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 14 02:50:30.680679 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.680046 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:30.684384 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.683644 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:30.684384 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:30.683728 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hcgk" podUID="153662a6-ddf9-444e-912b-4c6e1beba9d3" Sep 14 02:50:30.686424 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.685790 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rxxxt" Sep 14 02:50:30.690573 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.687718 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 14 02:50:30.690573 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.688046 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 14 02:50:30.690573 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.688341 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 14 02:50:30.690573 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.688611 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-s99n5\"" Sep 14 02:50:30.690573 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.690482 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 14 02:50:30.693321 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.690942 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6sn84" Sep 14 02:50:30.693321 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.691115 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 14 02:50:30.693321 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.691328 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 14 02:50:30.693321 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.691424 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pzw52\"" Sep 14 02:50:30.693321 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.691655 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 14 02:50:30.693321 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.691668 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 14 02:50:30.693321 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.691899 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tphq9\"" Sep 14 02:50:30.693321 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.691934 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 14 02:50:30.693321 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.692161 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 14 02:50:30.693321 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.692294 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fd9hg\"" Sep 14 02:50:30.693321 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.692402 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 14 02:50:30.693321 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.692585 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:30.693321 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:30.692645 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jrk97" podUID="441362ed-9e44-4875-9e60-ff99bf8fa8b1" Sep 14 02:50:30.693321 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.692656 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 14 02:50:30.693321 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.692862 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.693321 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.693310 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-host-var-lib-cni-multus\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.694195 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.693338 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-hostroot\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.694195 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.693357 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-var-lib-openvswitch\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.694195 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.693382 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-run-ovn\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.694195 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.693418 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-host-run-ovn-kubernetes\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.694195 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.693440 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-host-cni-bin\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.694195 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.693455 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-multus-conf-dir\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.694195 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.693470 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0c7a5c8-9307-432a-94bc-bdad16bee258-cni-binary-copy\") pod \"multus-additional-cni-plugins-fjn2n\" (UID: \"e0c7a5c8-9307-432a-94bc-bdad16bee258\") " pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:30.694195 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.693493 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwgdb\" (UniqueName: \"kubernetes.io/projected/e0c7a5c8-9307-432a-94bc-bdad16bee258-kube-api-access-nwgdb\") pod \"multus-additional-cni-plugins-fjn2n\" (UID: \"e0c7a5c8-9307-432a-94bc-bdad16bee258\") " pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:30.694195 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.693543 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-host-slash\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.694195 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.693567 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-etc-openvswitch\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.694195 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.693590 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-node-log\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.694195 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.693627 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-host-cni-netd\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.694195 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.693669 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.694195 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.693695 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58a26876-794d-43a3-9b28-d8375839bbcf-host\") pod \"node-ca-tjwgv\" (UID: \"58a26876-794d-43a3-9b28-d8375839bbcf\") " pod="openshift-image-registry/node-ca-tjwgv" Sep 14 02:50:30.694195 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.693718 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0c7a5c8-9307-432a-94bc-bdad16bee258-system-cni-dir\") pod \"multus-additional-cni-plugins-fjn2n\" (UID: \"e0c7a5c8-9307-432a-94bc-bdad16bee258\") " pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:30.694195 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.693749 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-run-openvswitch\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.695004 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.693782 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4369115a-4032-4f5a-a3e4-c02efe665617-ovn-node-metrics-cert\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.695004 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.693807 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4369115a-4032-4f5a-a3e4-c02efe665617-ovnkube-script-lib\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.695004 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.693874 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f5997c3-9c35-4171-aa1b-07f85bb77a17-cni-binary-copy\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.695004 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.693899 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-multus-socket-dir-parent\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.695004 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.693944 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-host-run-k8s-cni-cncf-io\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.695004 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694008 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-host-var-lib-cni-bin\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.695004 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694041 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0c7a5c8-9307-432a-94bc-bdad16bee258-cnibin\") pod \"multus-additional-cni-plugins-fjn2n\" (UID: \"e0c7a5c8-9307-432a-94bc-bdad16bee258\") " pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:30.695004 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694067 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-etc-kubernetes\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.695004 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694091 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0c7a5c8-9307-432a-94bc-bdad16bee258-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fjn2n\" (UID: \"e0c7a5c8-9307-432a-94bc-bdad16bee258\") " pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:30.695004 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694117 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e0c7a5c8-9307-432a-94bc-bdad16bee258-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fjn2n\" (UID: \"e0c7a5c8-9307-432a-94bc-bdad16bee258\") " pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:30.695004 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694182 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-host-kubelet\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.695004 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694206 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-cnibin\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.695004 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694232 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0f5997c3-9c35-4171-aa1b-07f85bb77a17-multus-daemon-config\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.695004 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694270 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-host-run-multus-certs\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.695004 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694300 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwrpm\" (UniqueName: \"kubernetes.io/projected/0f5997c3-9c35-4171-aa1b-07f85bb77a17-kube-api-access-bwrpm\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.695004 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694326 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-systemd-units\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.695844 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694351 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gk28\" (UniqueName: \"kubernetes.io/projected/58a26876-794d-43a3-9b28-d8375839bbcf-kube-api-access-2gk28\") pod \"node-ca-tjwgv\" (UID: \"58a26876-794d-43a3-9b28-d8375839bbcf\") " pod="openshift-image-registry/node-ca-tjwgv" Sep 14 02:50:30.695844 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694378 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-host-run-netns\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.695844 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694449 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e0c7a5c8-9307-432a-94bc-bdad16bee258-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fjn2n\" (UID: \"e0c7a5c8-9307-432a-94bc-bdad16bee258\") " pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:30.695844 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694473 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-host-run-netns\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.695844 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694496 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-log-socket\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.695844 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694555 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqz47\" (UniqueName: \"kubernetes.io/projected/4369115a-4032-4f5a-a3e4-c02efe665617-kube-api-access-nqz47\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.695844 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694704 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58a26876-794d-43a3-9b28-d8375839bbcf-serviceca\") pod \"node-ca-tjwgv\" (UID: \"58a26876-794d-43a3-9b28-d8375839bbcf\") " pod="openshift-image-registry/node-ca-tjwgv" Sep 14 02:50:30.695844 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694739 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-system-cni-dir\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.695844 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694778 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-host-var-lib-kubelet\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.695844 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694790 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:30.695844 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:30.694845 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq8rq" podUID="bc313df4-a60b-4df9-8d21-bb1141d7df64" Sep 14 02:50:30.695844 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694827 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-run-systemd\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.695844 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694893 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4369115a-4032-4f5a-a3e4-c02efe665617-ovnkube-config\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.695844 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694925 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4369115a-4032-4f5a-a3e4-c02efe665617-env-overrides\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.695844 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.694994 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-multus-cni-dir\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.698900 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.697241 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 14 02:50:30.698900 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.697675 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 14 02:50:30.698900 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.697692 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 14 02:50:30.698900 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.697721 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-72hdt\"" Sep 14 02:50:30.698900 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.698204 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 14 02:50:30.698900 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.698240 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:30.698900 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.698648 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7tlp9\"" Sep 14 02:50:30.701636 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.701599 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xgj42" Sep 14 02:50:30.702087 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.702056 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 14 02:50:30.702696 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.702465 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 14 02:50:30.702839 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.702822 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 14 02:50:30.703103 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.703087 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 14 02:50:30.703322 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.703306 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7lpxs\"" Sep 14 02:50:30.704962 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.704944 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 14 02:50:30.705503 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.705382 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 14 02:50:30.706198 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.706066 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ck67h\"" Sep 14 02:50:30.749331 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.748976 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.ec2.internal" event={"ID":"edcf5b83cf757b447b27e9d1ae1ebd2b","Type":"ContainerStarted","Data":"b757eb37e7fbd2219007aa6a6dda002f6c56fc511631a1d6526476862eb92201"} Sep 14 02:50:30.754373 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.754330 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.ec2.internal" event={"ID":"52af06832f4006a522bf3f805dd3bd2d","Type":"ContainerStarted","Data":"78a96944291859b63566d458617d63fa9237a0e8963ce64ea82da172ce878ea6"} Sep 14 02:50:30.759249 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.759225 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-n956x" Sep 14 02:50:30.773040 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.773019 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-n956x" Sep 14 02:50:30.779178 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.779161 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 14 02:50:30.795652 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.795626 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-host-var-lib-cni-multus\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.795766 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.795670 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc313df4-a60b-4df9-8d21-bb1141d7df64-kubelet-config\") pod \"global-pull-secret-syncer-mq8rq\" (UID: \"bc313df4-a60b-4df9-8d21-bb1141d7df64\") " pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:30.795766 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.795701 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f4515d2-186d-4c05-a47b-afebc4bd9c24-etc-tuned\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.795766 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.795728 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-run-ovn\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.795766 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.795752 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-host-var-lib-cni-multus\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.796037 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.795753 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-host-cni-bin\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.796037 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.795803 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-host-cni-bin\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.796037 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.795805 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-run-ovn\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.796037 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.795809 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-multus-conf-dir\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.796037 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.795845 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-multus-conf-dir\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.796037 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.795861 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06330f4e-f78f-49f6-8c53-9565d0364232-tmp-dir\") pod \"node-resolver-xgj42\" (UID: \"06330f4e-f78f-49f6-8c53-9565d0364232\") " pod="openshift-dns/node-resolver-xgj42" Sep 14 02:50:30.796037 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.795889 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g995w\" (UniqueName: \"kubernetes.io/projected/4f4515d2-186d-4c05-a47b-afebc4bd9c24-kube-api-access-g995w\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.796037 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.795917 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c4477e1c-f2aa-4ec6-b278-6027353ca267-socket-dir\") pod \"aws-ebs-csi-driver-node-zbtgx\" (UID: \"c4477e1c-f2aa-4ec6-b278-6027353ca267\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:30.796037 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.795942 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c4477e1c-f2aa-4ec6-b278-6027353ca267-device-dir\") pod \"aws-ebs-csi-driver-node-zbtgx\" (UID: \"c4477e1c-f2aa-4ec6-b278-6027353ca267\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:30.796037 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.795970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-host-run-multus-certs\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.796037 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.795995 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwrpm\" (UniqueName: \"kubernetes.io/projected/0f5997c3-9c35-4171-aa1b-07f85bb77a17-kube-api-access-bwrpm\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.796037 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796021 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-etc-openvswitch\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.796037 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796033 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-host-run-multus-certs\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.796691 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796047 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f27m4\" (UniqueName: \"kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4\") pod \"network-check-target-5hcgk\" (UID: \"153662a6-ddf9-444e-912b-4c6e1beba9d3\") " pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:30.796691 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796074 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-etc-sysctl-conf\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.796691 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796104 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0c7a5c8-9307-432a-94bc-bdad16bee258-system-cni-dir\") pod \"multus-additional-cni-plugins-fjn2n\" (UID: \"e0c7a5c8-9307-432a-94bc-bdad16bee258\") " pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:30.796691 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796131 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4369115a-4032-4f5a-a3e4-c02efe665617-ovn-node-metrics-cert\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.796691 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796156 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4369115a-4032-4f5a-a3e4-c02efe665617-ovnkube-script-lib\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.796691 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796184 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-multus-socket-dir-parent\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.796691 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796210 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-etc-systemd\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.796691 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796218 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0c7a5c8-9307-432a-94bc-bdad16bee258-system-cni-dir\") pod \"multus-additional-cni-plugins-fjn2n\" (UID: \"e0c7a5c8-9307-432a-94bc-bdad16bee258\") " pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:30.796691 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796237 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-etc-kubernetes\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.796691 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796262 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmh2n\" (UniqueName: \"kubernetes.io/projected/06330f4e-f78f-49f6-8c53-9565d0364232-kube-api-access-vmh2n\") pod \"node-resolver-xgj42\" (UID: \"06330f4e-f78f-49f6-8c53-9565d0364232\") " pod="openshift-dns/node-resolver-xgj42" Sep 14 02:50:30.796691 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796265 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-etc-openvswitch\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.796691 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796304 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0c7a5c8-9307-432a-94bc-bdad16bee258-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fjn2n\" (UID: \"e0c7a5c8-9307-432a-94bc-bdad16bee258\") " pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:30.796691 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796450 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e0c7a5c8-9307-432a-94bc-bdad16bee258-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fjn2n\" (UID: \"e0c7a5c8-9307-432a-94bc-bdad16bee258\") " pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:30.796691 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796502 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret\") pod \"global-pull-secret-syncer-mq8rq\" (UID: \"bc313df4-a60b-4df9-8d21-bb1141d7df64\") " pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:30.796691 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796536 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/50340084-82e7-48f8-93b5-2d99b73389af-iptables-alerter-script\") pod \"iptables-alerter-rxxxt\" (UID: \"50340084-82e7-48f8-93b5-2d99b73389af\") " pod="openshift-network-operator/iptables-alerter-rxxxt" Sep 14 02:50:30.796691 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796564 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-etc-kubernetes\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.797320 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796593 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gk28\" (UniqueName: \"kubernetes.io/projected/58a26876-794d-43a3-9b28-d8375839bbcf-kube-api-access-2gk28\") pod \"node-ca-tjwgv\" (UID: \"58a26876-794d-43a3-9b28-d8375839bbcf\") " pod="openshift-image-registry/node-ca-tjwgv" Sep 14 02:50:30.797320 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796622 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-host-run-netns\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.797320 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796647 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/06330f4e-f78f-49f6-8c53-9565d0364232-hosts-file\") pod \"node-resolver-xgj42\" (UID: \"06330f4e-f78f-49f6-8c53-9565d0364232\") " pod="openshift-dns/node-resolver-xgj42" Sep 14 02:50:30.797320 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796537 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0c7a5c8-9307-432a-94bc-bdad16bee258-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fjn2n\" (UID: \"e0c7a5c8-9307-432a-94bc-bdad16bee258\") " pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:30.797320 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796626 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-multus-socket-dir-parent\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.797320 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796678 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 14 02:50:30.797320 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796717 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-etc-kubernetes\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.797320 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796766 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-etc-modprobe-d\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.797320 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796794 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-etc-sysctl-d\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.797320 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796823 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e0c7a5c8-9307-432a-94bc-bdad16bee258-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fjn2n\" (UID: \"e0c7a5c8-9307-432a-94bc-bdad16bee258\") " pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:30.797320 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796851 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-host-run-netns\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.797320 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796858 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4369115a-4032-4f5a-a3e4-c02efe665617-ovnkube-script-lib\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.797320 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796912 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqz47\" (UniqueName: \"kubernetes.io/projected/4369115a-4032-4f5a-a3e4-c02efe665617-kube-api-access-nqz47\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.797320 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796932 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-host-run-netns\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.797320 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796939 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-system-cni-dir\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.797320 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796965 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-host-var-lib-kubelet\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.797320 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.796970 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-host-run-netns\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.797320 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797027 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-system-cni-dir\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.798234 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797018 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/48bf1d45-d3d3-4bde-8f44-688b868762d3-agent-certs\") pod \"konnectivity-agent-6sn84\" (UID: \"48bf1d45-d3d3-4bde-8f44-688b868762d3\") " pod="kube-system/konnectivity-agent-6sn84" Sep 14 02:50:30.798234 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797067 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-run\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.798234 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797077 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-host-var-lib-kubelet\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.798234 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797146 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-run-systemd\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.798234 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797181 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4369115a-4032-4f5a-a3e4-c02efe665617-env-overrides\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.798234 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797207 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c4477e1c-f2aa-4ec6-b278-6027353ca267-registration-dir\") pod \"aws-ebs-csi-driver-node-zbtgx\" (UID: \"c4477e1c-f2aa-4ec6-b278-6027353ca267\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:30.798234 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797205 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e0c7a5c8-9307-432a-94bc-bdad16bee258-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fjn2n\" (UID: \"e0c7a5c8-9307-432a-94bc-bdad16bee258\") " pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:30.798234 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797249 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c4477e1c-f2aa-4ec6-b278-6027353ca267-etc-selinux\") pod \"aws-ebs-csi-driver-node-zbtgx\" (UID: \"c4477e1c-f2aa-4ec6-b278-6027353ca267\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:30.798234 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797276 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c4477e1c-f2aa-4ec6-b278-6027353ca267-sys-fs\") pod \"aws-ebs-csi-driver-node-zbtgx\" (UID: \"c4477e1c-f2aa-4ec6-b278-6027353ca267\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:30.798234 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797322 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-var-lib-openvswitch\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.798234 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797319 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-run-systemd\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.798234 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797358 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-host-run-ovn-kubernetes\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.798234 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797390 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc313df4-a60b-4df9-8d21-bb1141d7df64-dbus\") pod \"global-pull-secret-syncer-mq8rq\" (UID: \"bc313df4-a60b-4df9-8d21-bb1141d7df64\") " pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:30.798234 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797394 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-host-run-ovn-kubernetes\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.798234 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797393 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-var-lib-openvswitch\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.798234 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797430 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/48bf1d45-d3d3-4bde-8f44-688b868762d3-konnectivity-ca\") pod \"konnectivity-agent-6sn84\" (UID: \"48bf1d45-d3d3-4bde-8f44-688b868762d3\") " pod="kube-system/konnectivity-agent-6sn84" Sep 14 02:50:30.798234 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797432 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e0c7a5c8-9307-432a-94bc-bdad16bee258-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fjn2n\" (UID: \"e0c7a5c8-9307-432a-94bc-bdad16bee258\") " pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:30.799108 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797457 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4s2l\" (UniqueName: \"kubernetes.io/projected/441362ed-9e44-4875-9e60-ff99bf8fa8b1-kube-api-access-n4s2l\") pod \"network-metrics-daemon-jrk97\" (UID: \"441362ed-9e44-4875-9e60-ff99bf8fa8b1\") " pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:30.799108 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797529 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0c7a5c8-9307-432a-94bc-bdad16bee258-cni-binary-copy\") pod \"multus-additional-cni-plugins-fjn2n\" (UID: \"e0c7a5c8-9307-432a-94bc-bdad16bee258\") " pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:30.799108 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwgdb\" (UniqueName: \"kubernetes.io/projected/e0c7a5c8-9307-432a-94bc-bdad16bee258-kube-api-access-nwgdb\") pod \"multus-additional-cni-plugins-fjn2n\" (UID: \"e0c7a5c8-9307-432a-94bc-bdad16bee258\") " pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:30.799108 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797578 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4369115a-4032-4f5a-a3e4-c02efe665617-env-overrides\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.799108 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797584 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-host-slash\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.799108 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797624 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-node-log\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.799108 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797650 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-host-cni-netd\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.799108 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797663 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-host-slash\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.799108 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797676 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.799108 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797702 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58a26876-794d-43a3-9b28-d8375839bbcf-host\") pod \"node-ca-tjwgv\" (UID: \"58a26876-794d-43a3-9b28-d8375839bbcf\") " pod="openshift-image-registry/node-ca-tjwgv" Sep 14 02:50:30.799108 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797710 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-node-log\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.799108 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797720 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-host-cni-netd\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.799108 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797759 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58a26876-794d-43a3-9b28-d8375839bbcf-host\") pod \"node-ca-tjwgv\" (UID: \"58a26876-794d-43a3-9b28-d8375839bbcf\") " pod="openshift-image-registry/node-ca-tjwgv" Sep 14 02:50:30.799108 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797770 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.799108 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797820 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f5997c3-9c35-4171-aa1b-07f85bb77a17-cni-binary-copy\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.799108 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797854 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c4477e1c-f2aa-4ec6-b278-6027353ca267-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zbtgx\" (UID: \"c4477e1c-f2aa-4ec6-b278-6027353ca267\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:30.799108 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797886 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-run-openvswitch\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.800043 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797914 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-host-run-k8s-cni-cncf-io\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.800043 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797940 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-host-var-lib-cni-bin\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.800043 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797962 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0c7a5c8-9307-432a-94bc-bdad16bee258-cni-binary-copy\") pod \"multus-additional-cni-plugins-fjn2n\" (UID: \"e0c7a5c8-9307-432a-94bc-bdad16bee258\") " pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:30.800043 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797980 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-run-openvswitch\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.800043 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.797990 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-host-var-lib-cni-bin\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.800043 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798018 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs\") pod \"network-metrics-daemon-jrk97\" (UID: \"441362ed-9e44-4875-9e60-ff99bf8fa8b1\") " pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:30.800043 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798028 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-host-run-k8s-cni-cncf-io\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.800043 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798049 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-etc-sysconfig\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.800043 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798073 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-sys\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.800043 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798125 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0c7a5c8-9307-432a-94bc-bdad16bee258-cnibin\") pod \"multus-additional-cni-plugins-fjn2n\" (UID: \"e0c7a5c8-9307-432a-94bc-bdad16bee258\") " pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:30.800043 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798154 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lknsw\" (UniqueName: \"kubernetes.io/projected/c4477e1c-f2aa-4ec6-b278-6027353ca267-kube-api-access-lknsw\") pod \"aws-ebs-csi-driver-node-zbtgx\" (UID: \"c4477e1c-f2aa-4ec6-b278-6027353ca267\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:30.800043 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798178 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50340084-82e7-48f8-93b5-2d99b73389af-host-slash\") pod \"iptables-alerter-rxxxt\" (UID: \"50340084-82e7-48f8-93b5-2d99b73389af\") " pod="openshift-network-operator/iptables-alerter-rxxxt" Sep 14 02:50:30.800043 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798196 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0c7a5c8-9307-432a-94bc-bdad16bee258-cnibin\") pod \"multus-additional-cni-plugins-fjn2n\" (UID: \"e0c7a5c8-9307-432a-94bc-bdad16bee258\") " pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:30.800043 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798202 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-host\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.800043 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798246 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-host-kubelet\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.800043 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-cnibin\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.800043 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-lib-modules\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.800899 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798323 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f4515d2-186d-4c05-a47b-afebc4bd9c24-tmp\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.800899 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798335 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-host-kubelet\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.800899 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798349 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-systemd-units\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.800899 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798375 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-var-lib-kubelet\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.800899 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798381 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-cnibin\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.800899 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798391 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f5997c3-9c35-4171-aa1b-07f85bb77a17-cni-binary-copy\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.800899 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0f5997c3-9c35-4171-aa1b-07f85bb77a17-multus-daemon-config\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.800899 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798423 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-systemd-units\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.800899 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798427 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-log-socket\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.800899 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798454 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58a26876-794d-43a3-9b28-d8375839bbcf-serviceca\") pod \"node-ca-tjwgv\" (UID: \"58a26876-794d-43a3-9b28-d8375839bbcf\") " pod="openshift-image-registry/node-ca-tjwgv" Sep 14 02:50:30.800899 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798479 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-hostroot\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.800899 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798524 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqpsp\" (UniqueName: \"kubernetes.io/projected/50340084-82e7-48f8-93b5-2d99b73389af-kube-api-access-xqpsp\") pod \"iptables-alerter-rxxxt\" (UID: \"50340084-82e7-48f8-93b5-2d99b73389af\") " pod="openshift-network-operator/iptables-alerter-rxxxt" Sep 14 02:50:30.800899 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798553 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4369115a-4032-4f5a-a3e4-c02efe665617-ovnkube-config\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.800899 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798579 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-multus-cni-dir\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.800899 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798601 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4369115a-4032-4f5a-a3e4-c02efe665617-log-socket\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.800899 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798606 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-hostroot\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.800899 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798784 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f5997c3-9c35-4171-aa1b-07f85bb77a17-multus-cni-dir\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.800899 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.798877 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0f5997c3-9c35-4171-aa1b-07f85bb77a17-multus-daemon-config\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.801846 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.799042 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58a26876-794d-43a3-9b28-d8375839bbcf-serviceca\") pod \"node-ca-tjwgv\" (UID: \"58a26876-794d-43a3-9b28-d8375839bbcf\") " pod="openshift-image-registry/node-ca-tjwgv" Sep 14 02:50:30.801846 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.799169 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4369115a-4032-4f5a-a3e4-c02efe665617-ovnkube-config\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.801846 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.801480 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4369115a-4032-4f5a-a3e4-c02efe665617-ovn-node-metrics-cert\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.810867 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.810799 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwrpm\" (UniqueName: \"kubernetes.io/projected/0f5997c3-9c35-4171-aa1b-07f85bb77a17-kube-api-access-bwrpm\") pod \"multus-n5mdt\" (UID: \"0f5997c3-9c35-4171-aa1b-07f85bb77a17\") " pod="openshift-multus/multus-n5mdt" Sep 14 02:50:30.813342 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.813078 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gk28\" (UniqueName: \"kubernetes.io/projected/58a26876-794d-43a3-9b28-d8375839bbcf-kube-api-access-2gk28\") pod \"node-ca-tjwgv\" (UID: \"58a26876-794d-43a3-9b28-d8375839bbcf\") " pod="openshift-image-registry/node-ca-tjwgv" Sep 14 02:50:30.813861 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.813837 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwgdb\" (UniqueName: \"kubernetes.io/projected/e0c7a5c8-9307-432a-94bc-bdad16bee258-kube-api-access-nwgdb\") pod \"multus-additional-cni-plugins-fjn2n\" (UID: \"e0c7a5c8-9307-432a-94bc-bdad16bee258\") " pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:30.816040 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.816012 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqz47\" (UniqueName: \"kubernetes.io/projected/4369115a-4032-4f5a-a3e4-c02efe665617-kube-api-access-nqz47\") pod \"ovnkube-node-5bwv6\" (UID: \"4369115a-4032-4f5a-a3e4-c02efe665617\") " pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:30.899230 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899203 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqpsp\" (UniqueName: \"kubernetes.io/projected/50340084-82e7-48f8-93b5-2d99b73389af-kube-api-access-xqpsp\") pod \"iptables-alerter-rxxxt\" (UID: \"50340084-82e7-48f8-93b5-2d99b73389af\") " pod="openshift-network-operator/iptables-alerter-rxxxt" Sep 14 02:50:30.899368 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899248 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc313df4-a60b-4df9-8d21-bb1141d7df64-kubelet-config\") pod \"global-pull-secret-syncer-mq8rq\" (UID: \"bc313df4-a60b-4df9-8d21-bb1141d7df64\") " pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:30.899368 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899280 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f4515d2-186d-4c05-a47b-afebc4bd9c24-etc-tuned\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.899368 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899309 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06330f4e-f78f-49f6-8c53-9565d0364232-tmp-dir\") pod \"node-resolver-xgj42\" (UID: \"06330f4e-f78f-49f6-8c53-9565d0364232\") " pod="openshift-dns/node-resolver-xgj42" Sep 14 02:50:30.899368 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899332 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g995w\" (UniqueName: \"kubernetes.io/projected/4f4515d2-186d-4c05-a47b-afebc4bd9c24-kube-api-access-g995w\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.899593 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899373 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c4477e1c-f2aa-4ec6-b278-6027353ca267-socket-dir\") pod \"aws-ebs-csi-driver-node-zbtgx\" (UID: \"c4477e1c-f2aa-4ec6-b278-6027353ca267\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:30.899593 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899395 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c4477e1c-f2aa-4ec6-b278-6027353ca267-device-dir\") pod \"aws-ebs-csi-driver-node-zbtgx\" (UID: \"c4477e1c-f2aa-4ec6-b278-6027353ca267\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:30.899593 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899414 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f27m4\" (UniqueName: \"kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4\") pod \"network-check-target-5hcgk\" (UID: \"153662a6-ddf9-444e-912b-4c6e1beba9d3\") " pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:30.899593 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899445 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-etc-sysctl-conf\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.899593 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899462 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-etc-systemd\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.899593 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899489 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmh2n\" (UniqueName: \"kubernetes.io/projected/06330f4e-f78f-49f6-8c53-9565d0364232-kube-api-access-vmh2n\") pod \"node-resolver-xgj42\" (UID: \"06330f4e-f78f-49f6-8c53-9565d0364232\") " pod="openshift-dns/node-resolver-xgj42" Sep 14 02:50:30.899593 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899528 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret\") pod \"global-pull-secret-syncer-mq8rq\" (UID: \"bc313df4-a60b-4df9-8d21-bb1141d7df64\") " pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:30.899593 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899577 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/50340084-82e7-48f8-93b5-2d99b73389af-iptables-alerter-script\") pod \"iptables-alerter-rxxxt\" (UID: \"50340084-82e7-48f8-93b5-2d99b73389af\") " pod="openshift-network-operator/iptables-alerter-rxxxt" Sep 14 02:50:30.899994 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899617 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-etc-kubernetes\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.899994 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899652 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/06330f4e-f78f-49f6-8c53-9565d0364232-hosts-file\") pod \"node-resolver-xgj42\" (UID: \"06330f4e-f78f-49f6-8c53-9565d0364232\") " pod="openshift-dns/node-resolver-xgj42" Sep 14 02:50:30.899994 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899675 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-etc-modprobe-d\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.899994 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899717 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-etc-sysctl-d\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.899994 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899753 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/48bf1d45-d3d3-4bde-8f44-688b868762d3-agent-certs\") pod \"konnectivity-agent-6sn84\" (UID: \"48bf1d45-d3d3-4bde-8f44-688b868762d3\") " pod="kube-system/konnectivity-agent-6sn84" Sep 14 02:50:30.899994 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899772 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-run\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.899994 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899796 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c4477e1c-f2aa-4ec6-b278-6027353ca267-registration-dir\") pod \"aws-ebs-csi-driver-node-zbtgx\" (UID: \"c4477e1c-f2aa-4ec6-b278-6027353ca267\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:30.899994 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899824 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c4477e1c-f2aa-4ec6-b278-6027353ca267-etc-selinux\") pod \"aws-ebs-csi-driver-node-zbtgx\" (UID: \"c4477e1c-f2aa-4ec6-b278-6027353ca267\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:30.899994 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899841 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c4477e1c-f2aa-4ec6-b278-6027353ca267-sys-fs\") pod \"aws-ebs-csi-driver-node-zbtgx\" (UID: \"c4477e1c-f2aa-4ec6-b278-6027353ca267\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:30.899994 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899841 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c4477e1c-f2aa-4ec6-b278-6027353ca267-socket-dir\") pod \"aws-ebs-csi-driver-node-zbtgx\" (UID: \"c4477e1c-f2aa-4ec6-b278-6027353ca267\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:30.899994 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899893 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc313df4-a60b-4df9-8d21-bb1141d7df64-dbus\") pod \"global-pull-secret-syncer-mq8rq\" (UID: \"bc313df4-a60b-4df9-8d21-bb1141d7df64\") " pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:30.899994 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899931 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/48bf1d45-d3d3-4bde-8f44-688b868762d3-konnectivity-ca\") pod \"konnectivity-agent-6sn84\" (UID: \"48bf1d45-d3d3-4bde-8f44-688b868762d3\") " pod="kube-system/konnectivity-agent-6sn84" Sep 14 02:50:30.899994 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.899964 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4s2l\" (UniqueName: \"kubernetes.io/projected/441362ed-9e44-4875-9e60-ff99bf8fa8b1-kube-api-access-n4s2l\") pod \"network-metrics-daemon-jrk97\" (UID: \"441362ed-9e44-4875-9e60-ff99bf8fa8b1\") " pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:30.899994 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900002 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c4477e1c-f2aa-4ec6-b278-6027353ca267-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zbtgx\" (UID: \"c4477e1c-f2aa-4ec6-b278-6027353ca267\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:30.900901 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900030 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs\") pod \"network-metrics-daemon-jrk97\" (UID: \"441362ed-9e44-4875-9e60-ff99bf8fa8b1\") " pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:30.900901 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900057 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-etc-sysconfig\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.900901 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900083 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-sys\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.900901 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900102 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lknsw\" (UniqueName: \"kubernetes.io/projected/c4477e1c-f2aa-4ec6-b278-6027353ca267-kube-api-access-lknsw\") pod \"aws-ebs-csi-driver-node-zbtgx\" (UID: \"c4477e1c-f2aa-4ec6-b278-6027353ca267\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:30.900901 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900117 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50340084-82e7-48f8-93b5-2d99b73389af-host-slash\") pod \"iptables-alerter-rxxxt\" (UID: \"50340084-82e7-48f8-93b5-2d99b73389af\") " pod="openshift-network-operator/iptables-alerter-rxxxt" Sep 14 02:50:30.900901 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900131 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-host\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.900901 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900158 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-lib-modules\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.900901 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900163 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06330f4e-f78f-49f6-8c53-9565d0364232-tmp-dir\") pod \"node-resolver-xgj42\" (UID: \"06330f4e-f78f-49f6-8c53-9565d0364232\") " pod="openshift-dns/node-resolver-xgj42" Sep 14 02:50:30.900901 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900178 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f4515d2-186d-4c05-a47b-afebc4bd9c24-tmp\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.900901 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900195 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-var-lib-kubelet\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.900901 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900243 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c4477e1c-f2aa-4ec6-b278-6027353ca267-sys-fs\") pod \"aws-ebs-csi-driver-node-zbtgx\" (UID: \"c4477e1c-f2aa-4ec6-b278-6027353ca267\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:30.900901 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900274 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-var-lib-kubelet\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.900901 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900266 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-etc-sysctl-conf\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.900901 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900309 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c4477e1c-f2aa-4ec6-b278-6027353ca267-device-dir\") pod \"aws-ebs-csi-driver-node-zbtgx\" (UID: \"c4477e1c-f2aa-4ec6-b278-6027353ca267\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:30.900901 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900337 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc313df4-a60b-4df9-8d21-bb1141d7df64-dbus\") pod \"global-pull-secret-syncer-mq8rq\" (UID: \"bc313df4-a60b-4df9-8d21-bb1141d7df64\") " pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:30.900901 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900366 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-etc-systemd\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.900901 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900383 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc313df4-a60b-4df9-8d21-bb1141d7df64-kubelet-config\") pod \"global-pull-secret-syncer-mq8rq\" (UID: \"bc313df4-a60b-4df9-8d21-bb1141d7df64\") " pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:30.900901 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900500 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-sys\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.901856 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900513 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c4477e1c-f2aa-4ec6-b278-6027353ca267-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zbtgx\" (UID: \"c4477e1c-f2aa-4ec6-b278-6027353ca267\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:30.901856 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:30.900623 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 02:50:30.901856 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900684 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-etc-kubernetes\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.901856 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900695 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-lib-modules\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.901856 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900794 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/06330f4e-f78f-49f6-8c53-9565d0364232-hosts-file\") pod \"node-resolver-xgj42\" (UID: \"06330f4e-f78f-49f6-8c53-9565d0364232\") " pod="openshift-dns/node-resolver-xgj42" Sep 14 02:50:30.901856 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900802 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-etc-modprobe-d\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.901856 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:30.900818 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs podName:441362ed-9e44-4875-9e60-ff99bf8fa8b1 nodeName:}" failed. No retries permitted until 2026-09-14 02:50:31.400771355 +0000 UTC m=+3.130430452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs") pod "network-metrics-daemon-jrk97" (UID: "441362ed-9e44-4875-9e60-ff99bf8fa8b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 02:50:30.901856 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900930 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-etc-sysctl-d\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.901856 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900983 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/48bf1d45-d3d3-4bde-8f44-688b868762d3-konnectivity-ca\") pod \"konnectivity-agent-6sn84\" (UID: \"48bf1d45-d3d3-4bde-8f44-688b868762d3\") " pod="kube-system/konnectivity-agent-6sn84" Sep 14 02:50:30.901856 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.900901 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c4477e1c-f2aa-4ec6-b278-6027353ca267-etc-selinux\") pod \"aws-ebs-csi-driver-node-zbtgx\" (UID: \"c4477e1c-f2aa-4ec6-b278-6027353ca267\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:30.901856 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.901064 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50340084-82e7-48f8-93b5-2d99b73389af-host-slash\") pod \"iptables-alerter-rxxxt\" (UID: \"50340084-82e7-48f8-93b5-2d99b73389af\") " pod="openshift-network-operator/iptables-alerter-rxxxt" Sep 14 02:50:30.901856 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.901137 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c4477e1c-f2aa-4ec6-b278-6027353ca267-registration-dir\") pod \"aws-ebs-csi-driver-node-zbtgx\" (UID: \"c4477e1c-f2aa-4ec6-b278-6027353ca267\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:30.901856 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.901169 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-etc-sysconfig\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.901856 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:30.901192 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 02:50:30.901856 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.901199 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-run\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.901856 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:30.901295 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret podName:bc313df4-a60b-4df9-8d21-bb1141d7df64 nodeName:}" failed. No retries permitted until 2026-09-14 02:50:31.401276593 +0000 UTC m=+3.130935691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret") pod "global-pull-secret-syncer-mq8rq" (UID: "bc313df4-a60b-4df9-8d21-bb1141d7df64") : object "kube-system"/"original-pull-secret" not registered Sep 14 02:50:30.901856 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.901329 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/50340084-82e7-48f8-93b5-2d99b73389af-iptables-alerter-script\") pod \"iptables-alerter-rxxxt\" (UID: \"50340084-82e7-48f8-93b5-2d99b73389af\") " pod="openshift-network-operator/iptables-alerter-rxxxt" Sep 14 02:50:30.902524 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.901354 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f4515d2-186d-4c05-a47b-afebc4bd9c24-host\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.902524 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.902433 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4f4515d2-186d-4c05-a47b-afebc4bd9c24-etc-tuned\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.903925 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.903110 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f4515d2-186d-4c05-a47b-afebc4bd9c24-tmp\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.904092 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.904070 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/48bf1d45-d3d3-4bde-8f44-688b868762d3-agent-certs\") pod \"konnectivity-agent-6sn84\" (UID: \"48bf1d45-d3d3-4bde-8f44-688b868762d3\") " pod="kube-system/konnectivity-agent-6sn84" Sep 14 02:50:30.918513 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:30.918442 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 02:50:30.918513 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:30.918463 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 02:50:30.918513 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:30.918477 2564 projected.go:196] Error preparing data for projected volume kube-api-access-f27m4 for pod openshift-network-diagnostics/network-check-target-5hcgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 02:50:30.918722 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:30.918561 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4 podName:153662a6-ddf9-444e-912b-4c6e1beba9d3 nodeName:}" failed. No retries permitted until 2026-09-14 02:50:31.418532505 +0000 UTC m=+3.148191602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f27m4" (UniqueName: "kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4") pod "network-check-target-5hcgk" (UID: "153662a6-ddf9-444e-912b-4c6e1beba9d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 02:50:30.921159 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.921128 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqpsp\" (UniqueName: \"kubernetes.io/projected/50340084-82e7-48f8-93b5-2d99b73389af-kube-api-access-xqpsp\") pod \"iptables-alerter-rxxxt\" (UID: \"50340084-82e7-48f8-93b5-2d99b73389af\") " pod="openshift-network-operator/iptables-alerter-rxxxt" Sep 14 02:50:30.921338 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.921313 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lknsw\" (UniqueName: \"kubernetes.io/projected/c4477e1c-f2aa-4ec6-b278-6027353ca267-kube-api-access-lknsw\") pod \"aws-ebs-csi-driver-node-zbtgx\" (UID: \"c4477e1c-f2aa-4ec6-b278-6027353ca267\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:30.921648 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.921623 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmh2n\" (UniqueName: \"kubernetes.io/projected/06330f4e-f78f-49f6-8c53-9565d0364232-kube-api-access-vmh2n\") pod \"node-resolver-xgj42\" (UID: \"06330f4e-f78f-49f6-8c53-9565d0364232\") " pod="openshift-dns/node-resolver-xgj42" Sep 14 02:50:30.922719 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.922697 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g995w\" (UniqueName: \"kubernetes.io/projected/4f4515d2-186d-4c05-a47b-afebc4bd9c24-kube-api-access-g995w\") pod \"tuned-67g56\" (UID: \"4f4515d2-186d-4c05-a47b-afebc4bd9c24\") " pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:30.929131 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.928717 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4s2l\" (UniqueName: \"kubernetes.io/projected/441362ed-9e44-4875-9e60-ff99bf8fa8b1-kube-api-access-n4s2l\") pod \"network-metrics-daemon-jrk97\" (UID: \"441362ed-9e44-4875-9e60-ff99bf8fa8b1\") " pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:30.987907 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:30.987878 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:31.003634 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:31.003611 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tjwgv" Sep 14 02:50:31.013808 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:31.013788 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n5mdt" Sep 14 02:50:31.020276 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:31.020247 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fjn2n" Sep 14 02:50:31.032221 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:31.032199 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rxxxt" Sep 14 02:50:31.041199 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:31.041181 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6sn84" Sep 14 02:50:31.052611 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:31.052590 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-67g56" Sep 14 02:50:31.059031 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:31.059009 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" Sep 14 02:50:31.066051 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:31.066034 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xgj42" Sep 14 02:50:31.404055 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:31.403982 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs\") pod \"network-metrics-daemon-jrk97\" (UID: \"441362ed-9e44-4875-9e60-ff99bf8fa8b1\") " pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:31.404194 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:31.404061 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret\") pod \"global-pull-secret-syncer-mq8rq\" (UID: \"bc313df4-a60b-4df9-8d21-bb1141d7df64\") " pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:31.404194 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:31.404158 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 02:50:31.404194 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:31.404165 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 02:50:31.404311 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:31.404227 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret podName:bc313df4-a60b-4df9-8d21-bb1141d7df64 nodeName:}" failed. No retries permitted until 2026-09-14 02:50:32.404207694 +0000 UTC m=+4.133866805 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret") pod "global-pull-secret-syncer-mq8rq" (UID: "bc313df4-a60b-4df9-8d21-bb1141d7df64") : object "kube-system"/"original-pull-secret" not registered Sep 14 02:50:31.404311 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:31.404244 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs podName:441362ed-9e44-4875-9e60-ff99bf8fa8b1 nodeName:}" failed. No retries permitted until 2026-09-14 02:50:32.404234916 +0000 UTC m=+4.133894013 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs") pod "network-metrics-daemon-jrk97" (UID: "441362ed-9e44-4875-9e60-ff99bf8fa8b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 02:50:31.504736 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:31.504704 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f27m4\" (UniqueName: \"kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4\") pod \"network-check-target-5hcgk\" (UID: \"153662a6-ddf9-444e-912b-4c6e1beba9d3\") " pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:31.504892 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:31.504856 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 02:50:31.504892 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:31.504877 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 02:50:31.504892 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:31.504886 2564 projected.go:196] Error preparing data for projected volume kube-api-access-f27m4 for pod openshift-network-diagnostics/network-check-target-5hcgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 02:50:31.505043 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:31.504952 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4 podName:153662a6-ddf9-444e-912b-4c6e1beba9d3 nodeName:}" failed. No retries permitted until 2026-09-14 02:50:32.504931418 +0000 UTC m=+4.234590516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f27m4" (UniqueName: "kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4") pod "network-check-target-5hcgk" (UID: "153662a6-ddf9-444e-912b-4c6e1beba9d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 02:50:31.776652 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:31.776563 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 02:45:30 +0000 UTC" deadline="2028-06-11 00:09:14.070576918 +0000 UTC" Sep 14 02:50:31.776652 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:31.776605 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15261h18m42.29397679s" Sep 14 02:50:32.110401 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:32.110375 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4369115a_4032_4f5a_a3e4_c02efe665617.slice/crio-072b30e6b961581501bdbc80ee94b5cdacf2c8ebfc058d920df4e582459d8228 WatchSource:0}: Error finding container 072b30e6b961581501bdbc80ee94b5cdacf2c8ebfc058d920df4e582459d8228: Status 404 returned error can't find the container with id 072b30e6b961581501bdbc80ee94b5cdacf2c8ebfc058d920df4e582459d8228 Sep 14 02:50:32.128069 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:32.128044 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4477e1c_f2aa_4ec6_b278_6027353ca267.slice/crio-827fddb29cb53ad0a6f128952fef2785d51b31741d9a923960e8a89411bb9c51 WatchSource:0}: Error finding container 827fddb29cb53ad0a6f128952fef2785d51b31741d9a923960e8a89411bb9c51: Status 404 returned error can't find the container with id 827fddb29cb53ad0a6f128952fef2785d51b31741d9a923960e8a89411bb9c51 Sep 14 02:50:32.129913 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:32.129889 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c7a5c8_9307_432a_94bc_bdad16bee258.slice/crio-284c08d483fb03b9c81f0b9323791271b5aa6c0303b8644f3fac42ee385c1817 WatchSource:0}: Error finding container 284c08d483fb03b9c81f0b9323791271b5aa6c0303b8644f3fac42ee385c1817: Status 404 returned error can't find the container with id 284c08d483fb03b9c81f0b9323791271b5aa6c0303b8644f3fac42ee385c1817 Sep 14 02:50:32.130351 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:32.130319 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50340084_82e7_48f8_93b5_2d99b73389af.slice/crio-6db9daaedd2a8fc330f358a06ef3d7232f9e06ee88f6b4b7f6e37b8f9c4e697d WatchSource:0}: Error finding container 6db9daaedd2a8fc330f358a06ef3d7232f9e06ee88f6b4b7f6e37b8f9c4e697d: Status 404 returned error can't find the container with id 6db9daaedd2a8fc330f358a06ef3d7232f9e06ee88f6b4b7f6e37b8f9c4e697d Sep 14 02:50:32.131271 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:32.131250 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f5997c3_9c35_4171_aa1b_07f85bb77a17.slice/crio-cc5999cff4d5a69c3a2c361fb2728c4b6e4f686b1c3e2ce7aed763aeabf400ac WatchSource:0}: Error finding container cc5999cff4d5a69c3a2c361fb2728c4b6e4f686b1c3e2ce7aed763aeabf400ac: Status 404 returned error can't find the container with id cc5999cff4d5a69c3a2c361fb2728c4b6e4f686b1c3e2ce7aed763aeabf400ac Sep 14 02:50:32.133365 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:32.133341 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06330f4e_f78f_49f6_8c53_9565d0364232.slice/crio-095e990de1c768ced2e55e0cee691a1c8eb109143a40c4eec677940bf30848b5 WatchSource:0}: Error finding container 095e990de1c768ced2e55e0cee691a1c8eb109143a40c4eec677940bf30848b5: Status 404 returned error can't find the container with id 095e990de1c768ced2e55e0cee691a1c8eb109143a40c4eec677940bf30848b5 Sep 14 02:50:32.134617 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:32.134591 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a26876_794d_43a3_9b28_d8375839bbcf.slice/crio-de70c1b519b027c045bb8789bb94dbad4786b606ea990a22a713bb2b6092510c WatchSource:0}: Error finding container de70c1b519b027c045bb8789bb94dbad4786b606ea990a22a713bb2b6092510c: Status 404 returned error can't find the container with id de70c1b519b027c045bb8789bb94dbad4786b606ea990a22a713bb2b6092510c Sep 14 02:50:32.135539 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:50:32.135493 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48bf1d45_d3d3_4bde_8f44_688b868762d3.slice/crio-a4b283fcaf9fb8573f7749f7c92721c19f5fbd586c65fb53d1e1b20818820c04 WatchSource:0}: Error finding container a4b283fcaf9fb8573f7749f7c92721c19f5fbd586c65fb53d1e1b20818820c04: Status 404 returned error can't find the container with id a4b283fcaf9fb8573f7749f7c92721c19f5fbd586c65fb53d1e1b20818820c04 Sep 14 02:50:32.411745 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:32.411724 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret\") pod \"global-pull-secret-syncer-mq8rq\" (UID: \"bc313df4-a60b-4df9-8d21-bb1141d7df64\") " pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:32.411831 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:32.411764 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs\") pod \"network-metrics-daemon-jrk97\" (UID: \"441362ed-9e44-4875-9e60-ff99bf8fa8b1\") " pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:32.411887 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:32.411871 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 02:50:32.411887 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:32.411878 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 02:50:32.411944 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:32.411933 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret podName:bc313df4-a60b-4df9-8d21-bb1141d7df64 nodeName:}" failed. No retries permitted until 2026-09-14 02:50:34.411916805 +0000 UTC m=+6.141575902 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret") pod "global-pull-secret-syncer-mq8rq" (UID: "bc313df4-a60b-4df9-8d21-bb1141d7df64") : object "kube-system"/"original-pull-secret" not registered Sep 14 02:50:32.411984 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:32.411950 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs podName:441362ed-9e44-4875-9e60-ff99bf8fa8b1 nodeName:}" failed. No retries permitted until 2026-09-14 02:50:34.411941093 +0000 UTC m=+6.141600188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs") pod "network-metrics-daemon-jrk97" (UID: "441362ed-9e44-4875-9e60-ff99bf8fa8b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 02:50:32.512514 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:32.512471 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f27m4\" (UniqueName: \"kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4\") pod \"network-check-target-5hcgk\" (UID: \"153662a6-ddf9-444e-912b-4c6e1beba9d3\") " pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:32.512669 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:32.512647 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 02:50:32.512669 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:32.512668 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 02:50:32.512774 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:32.512680 2564 projected.go:196] Error preparing data for projected volume kube-api-access-f27m4 for pod openshift-network-diagnostics/network-check-target-5hcgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 02:50:32.512774 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:32.512739 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4 podName:153662a6-ddf9-444e-912b-4c6e1beba9d3 nodeName:}" failed. No retries permitted until 2026-09-14 02:50:34.512721001 +0000 UTC m=+6.242380111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f27m4" (UniqueName: "kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4") pod "network-check-target-5hcgk" (UID: "153662a6-ddf9-444e-912b-4c6e1beba9d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 02:50:32.743935 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:32.743828 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:32.743935 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:32.743870 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:32.743935 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:32.743873 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:32.744193 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:32.743976 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jrk97" podUID="441362ed-9e44-4875-9e60-ff99bf8fa8b1" Sep 14 02:50:32.744193 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:32.744108 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq8rq" podUID="bc313df4-a60b-4df9-8d21-bb1141d7df64" Sep 14 02:50:32.750638 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:32.750600 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hcgk" podUID="153662a6-ddf9-444e-912b-4c6e1beba9d3" Sep 14 02:50:32.762536 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:32.762484 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n5mdt" event={"ID":"0f5997c3-9c35-4171-aa1b-07f85bb77a17","Type":"ContainerStarted","Data":"cc5999cff4d5a69c3a2c361fb2728c4b6e4f686b1c3e2ce7aed763aeabf400ac"} Sep 14 02:50:32.764598 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:32.764534 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjn2n" event={"ID":"e0c7a5c8-9307-432a-94bc-bdad16bee258","Type":"ContainerStarted","Data":"284c08d483fb03b9c81f0b9323791271b5aa6c0303b8644f3fac42ee385c1817"} Sep 14 02:50:32.766127 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:32.766103 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" event={"ID":"c4477e1c-f2aa-4ec6-b278-6027353ca267","Type":"ContainerStarted","Data":"827fddb29cb53ad0a6f128952fef2785d51b31741d9a923960e8a89411bb9c51"} Sep 14 02:50:32.768731 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:32.768640 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" event={"ID":"4369115a-4032-4f5a-a3e4-c02efe665617","Type":"ContainerStarted","Data":"072b30e6b961581501bdbc80ee94b5cdacf2c8ebfc058d920df4e582459d8228"} Sep 14 02:50:32.776986 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:32.776906 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 02:45:30 +0000 UTC" deadline="2028-03-12 00:31:55.76754643 +0000 UTC" Sep 14 02:50:32.776986 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:32.776953 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13077h41m22.990597327s" Sep 14 02:50:32.777569 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:32.777269 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.ec2.internal" event={"ID":"edcf5b83cf757b447b27e9d1ae1ebd2b","Type":"ContainerStarted","Data":"919683cfc8f49a17695cd797867cee102096ebd42ad7dff95944f54cb37d12ca"} Sep 14 02:50:32.779469 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:32.779352 2564 generic.go:356] "Generic (PLEG): container finished" podID="52af06832f4006a522bf3f805dd3bd2d" containerID="4cd6e946384510d325756c7566df11edf9c9eb99a328438fc4dc6e69add24dc0" exitCode=0 Sep 14 02:50:32.779469 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:32.779421 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.ec2.internal" event={"ID":"52af06832f4006a522bf3f805dd3bd2d","Type":"ContainerDied","Data":"4cd6e946384510d325756c7566df11edf9c9eb99a328438fc4dc6e69add24dc0"} Sep 14 02:50:32.782666 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:32.782640 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6sn84" event={"ID":"48bf1d45-d3d3-4bde-8f44-688b868762d3","Type":"ContainerStarted","Data":"a4b283fcaf9fb8573f7749f7c92721c19f5fbd586c65fb53d1e1b20818820c04"} Sep 14 02:50:32.784420 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:32.784397 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tjwgv" event={"ID":"58a26876-794d-43a3-9b28-d8375839bbcf","Type":"ContainerStarted","Data":"de70c1b519b027c045bb8789bb94dbad4786b606ea990a22a713bb2b6092510c"} Sep 14 02:50:32.786614 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:32.786589 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xgj42" event={"ID":"06330f4e-f78f-49f6-8c53-9565d0364232","Type":"ContainerStarted","Data":"095e990de1c768ced2e55e0cee691a1c8eb109143a40c4eec677940bf30848b5"} Sep 14 02:50:32.789190 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:32.789165 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rxxxt" event={"ID":"50340084-82e7-48f8-93b5-2d99b73389af","Type":"ContainerStarted","Data":"6db9daaedd2a8fc330f358a06ef3d7232f9e06ee88f6b4b7f6e37b8f9c4e697d"} Sep 14 02:50:32.794574 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:32.794515 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-67g56" event={"ID":"4f4515d2-186d-4c05-a47b-afebc4bd9c24","Type":"ContainerStarted","Data":"59b77a497f53ea2a3d66abe420415af80c912b8f880fdda34fc2198c5956624b"} Sep 14 02:50:32.859243 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:32.859188 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.ec2.internal" podStartSLOduration=2.859165536 podStartE2EDuration="2.859165536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:50:32.825947059 +0000 UTC m=+4.555606178" watchObservedRunningTime="2026-09-14 02:50:32.859165536 +0000 UTC m=+4.588824652" Sep 14 02:50:33.828606 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:33.828537 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.ec2.internal" event={"ID":"52af06832f4006a522bf3f805dd3bd2d","Type":"ContainerStarted","Data":"69e0b53dcec72add0652e6273a5283686e063bc679891deeb0ed4a19e0fe4e00"} Sep 14 02:50:33.854743 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:33.854692 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.ec2.internal" podStartSLOduration=3.854672506 podStartE2EDuration="3.854672506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:50:33.853740544 +0000 UTC m=+5.583399661" watchObservedRunningTime="2026-09-14 02:50:33.854672506 +0000 UTC m=+5.584331649" Sep 14 02:50:34.429460 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:34.429407 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret\") pod \"global-pull-secret-syncer-mq8rq\" (UID: \"bc313df4-a60b-4df9-8d21-bb1141d7df64\") " pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:34.429665 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:34.429509 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs\") pod \"network-metrics-daemon-jrk97\" (UID: \"441362ed-9e44-4875-9e60-ff99bf8fa8b1\") " pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:34.429665 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:34.429638 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 02:50:34.429767 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:34.429695 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs podName:441362ed-9e44-4875-9e60-ff99bf8fa8b1 nodeName:}" failed. No retries permitted until 2026-09-14 02:50:38.429677242 +0000 UTC m=+10.159336341 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs") pod "network-metrics-daemon-jrk97" (UID: "441362ed-9e44-4875-9e60-ff99bf8fa8b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 02:50:34.430091 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:34.430068 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 02:50:34.430169 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:34.430128 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret podName:bc313df4-a60b-4df9-8d21-bb1141d7df64 nodeName:}" failed. No retries permitted until 2026-09-14 02:50:38.43011347 +0000 UTC m=+10.159772568 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret") pod "global-pull-secret-syncer-mq8rq" (UID: "bc313df4-a60b-4df9-8d21-bb1141d7df64") : object "kube-system"/"original-pull-secret" not registered Sep 14 02:50:34.530085 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:34.530050 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f27m4\" (UniqueName: \"kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4\") pod \"network-check-target-5hcgk\" (UID: \"153662a6-ddf9-444e-912b-4c6e1beba9d3\") " pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:34.530253 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:34.530220 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 02:50:34.530253 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:34.530241 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 02:50:34.530253 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:34.530250 2564 projected.go:196] Error preparing data for projected volume kube-api-access-f27m4 for pod openshift-network-diagnostics/network-check-target-5hcgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 02:50:34.530414 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:34.530302 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4 podName:153662a6-ddf9-444e-912b-4c6e1beba9d3 nodeName:}" failed. No retries permitted until 2026-09-14 02:50:38.530284171 +0000 UTC m=+10.259943281 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f27m4" (UniqueName: "kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4") pod "network-check-target-5hcgk" (UID: "153662a6-ddf9-444e-912b-4c6e1beba9d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 02:50:34.746932 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:34.746235 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:34.746932 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:34.746407 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq8rq" podUID="bc313df4-a60b-4df9-8d21-bb1141d7df64" Sep 14 02:50:34.746932 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:34.746643 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:34.746932 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:34.746883 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jrk97" podUID="441362ed-9e44-4875-9e60-ff99bf8fa8b1" Sep 14 02:50:34.746932 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:34.746932 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:34.747295 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:34.747007 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hcgk" podUID="153662a6-ddf9-444e-912b-4c6e1beba9d3" Sep 14 02:50:36.746078 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:36.745405 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:36.746078 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:36.745517 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hcgk" podUID="153662a6-ddf9-444e-912b-4c6e1beba9d3" Sep 14 02:50:36.746078 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:36.745847 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:36.746078 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:36.745929 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jrk97" podUID="441362ed-9e44-4875-9e60-ff99bf8fa8b1" Sep 14 02:50:36.746078 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:36.745975 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:36.746078 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:36.746033 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq8rq" podUID="bc313df4-a60b-4df9-8d21-bb1141d7df64" Sep 14 02:50:38.464326 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:38.463611 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs\") pod \"network-metrics-daemon-jrk97\" (UID: \"441362ed-9e44-4875-9e60-ff99bf8fa8b1\") " pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:38.464326 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:38.463690 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret\") pod \"global-pull-secret-syncer-mq8rq\" (UID: \"bc313df4-a60b-4df9-8d21-bb1141d7df64\") " pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:38.464326 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:38.463817 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 02:50:38.464326 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:38.463873 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret podName:bc313df4-a60b-4df9-8d21-bb1141d7df64 nodeName:}" failed. No retries permitted until 2026-09-14 02:50:46.4638545 +0000 UTC m=+18.193513602 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret") pod "global-pull-secret-syncer-mq8rq" (UID: "bc313df4-a60b-4df9-8d21-bb1141d7df64") : object "kube-system"/"original-pull-secret" not registered Sep 14 02:50:38.464326 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:38.464247 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 02:50:38.464326 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:38.464296 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs podName:441362ed-9e44-4875-9e60-ff99bf8fa8b1 nodeName:}" failed. No retries permitted until 2026-09-14 02:50:46.464279717 +0000 UTC m=+18.193938816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs") pod "network-metrics-daemon-jrk97" (UID: "441362ed-9e44-4875-9e60-ff99bf8fa8b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 02:50:38.564410 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:38.564374 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f27m4\" (UniqueName: \"kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4\") pod \"network-check-target-5hcgk\" (UID: \"153662a6-ddf9-444e-912b-4c6e1beba9d3\") " pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:38.564616 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:38.564558 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 02:50:38.564616 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:38.564579 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 02:50:38.564616 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:38.564591 2564 projected.go:196] Error preparing data for projected volume kube-api-access-f27m4 for pod openshift-network-diagnostics/network-check-target-5hcgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 02:50:38.564774 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:38.564640 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4 podName:153662a6-ddf9-444e-912b-4c6e1beba9d3 nodeName:}" failed. No retries permitted until 2026-09-14 02:50:46.564623791 +0000 UTC m=+18.294282902 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f27m4" (UniqueName: "kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4") pod "network-check-target-5hcgk" (UID: "153662a6-ddf9-444e-912b-4c6e1beba9d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 02:50:38.748983 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:38.748608 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:38.748983 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:38.748788 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:38.748983 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:38.748808 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:38.748983 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:38.748823 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jrk97" podUID="441362ed-9e44-4875-9e60-ff99bf8fa8b1" Sep 14 02:50:38.748983 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:38.748888 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hcgk" podUID="153662a6-ddf9-444e-912b-4c6e1beba9d3" Sep 14 02:50:38.748983 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:38.748965 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq8rq" podUID="bc313df4-a60b-4df9-8d21-bb1141d7df64" Sep 14 02:50:40.744699 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:40.744608 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:40.745073 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:40.744726 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq8rq" podUID="bc313df4-a60b-4df9-8d21-bb1141d7df64" Sep 14 02:50:40.745073 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:40.745061 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:40.745187 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:40.745143 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hcgk" podUID="153662a6-ddf9-444e-912b-4c6e1beba9d3" Sep 14 02:50:40.745296 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:40.745281 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:40.745392 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:40.745373 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jrk97" podUID="441362ed-9e44-4875-9e60-ff99bf8fa8b1" Sep 14 02:50:42.744421 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:42.744331 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:42.744421 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:42.744363 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:42.744421 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:42.744339 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:42.744974 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:42.744433 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hcgk" podUID="153662a6-ddf9-444e-912b-4c6e1beba9d3" Sep 14 02:50:42.744974 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:42.744564 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jrk97" podUID="441362ed-9e44-4875-9e60-ff99bf8fa8b1" Sep 14 02:50:42.744974 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:42.744658 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq8rq" podUID="bc313df4-a60b-4df9-8d21-bb1141d7df64" Sep 14 02:50:44.744628 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:44.744577 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:44.745109 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:44.744597 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:44.745109 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:44.744730 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jrk97" podUID="441362ed-9e44-4875-9e60-ff99bf8fa8b1" Sep 14 02:50:44.745109 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:44.744804 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq8rq" podUID="bc313df4-a60b-4df9-8d21-bb1141d7df64" Sep 14 02:50:44.745109 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:44.744821 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:44.745109 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:44.744903 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hcgk" podUID="153662a6-ddf9-444e-912b-4c6e1beba9d3" Sep 14 02:50:46.526518 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:46.526463 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret\") pod \"global-pull-secret-syncer-mq8rq\" (UID: \"bc313df4-a60b-4df9-8d21-bb1141d7df64\") " pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:46.526941 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:46.526572 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs\") pod \"network-metrics-daemon-jrk97\" (UID: \"441362ed-9e44-4875-9e60-ff99bf8fa8b1\") " pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:46.526941 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:46.526628 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 02:50:46.526941 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:46.526687 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret podName:bc313df4-a60b-4df9-8d21-bb1141d7df64 nodeName:}" failed. No retries permitted until 2026-09-14 02:51:02.526673568 +0000 UTC m=+34.256332667 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret") pod "global-pull-secret-syncer-mq8rq" (UID: "bc313df4-a60b-4df9-8d21-bb1141d7df64") : object "kube-system"/"original-pull-secret" not registered Sep 14 02:50:46.526941 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:46.526737 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 02:50:46.526941 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:46.526804 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs podName:441362ed-9e44-4875-9e60-ff99bf8fa8b1 nodeName:}" failed. No retries permitted until 2026-09-14 02:51:02.526785525 +0000 UTC m=+34.256444621 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs") pod "network-metrics-daemon-jrk97" (UID: "441362ed-9e44-4875-9e60-ff99bf8fa8b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 02:50:46.627082 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:46.626980 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f27m4\" (UniqueName: \"kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4\") pod \"network-check-target-5hcgk\" (UID: \"153662a6-ddf9-444e-912b-4c6e1beba9d3\") " pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:46.627283 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:46.627117 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 02:50:46.627283 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:46.627135 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 02:50:46.627283 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:46.627148 2564 projected.go:196] Error preparing data for projected volume kube-api-access-f27m4 for pod openshift-network-diagnostics/network-check-target-5hcgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 02:50:46.627283 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:46.627211 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4 podName:153662a6-ddf9-444e-912b-4c6e1beba9d3 nodeName:}" failed. No retries permitted until 2026-09-14 02:51:02.627192829 +0000 UTC m=+34.356851930 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f27m4" (UniqueName: "kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4") pod "network-check-target-5hcgk" (UID: "153662a6-ddf9-444e-912b-4c6e1beba9d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 02:50:46.743579 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:46.743539 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:46.743745 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:46.743539 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:46.743745 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:46.743673 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq8rq" podUID="bc313df4-a60b-4df9-8d21-bb1141d7df64" Sep 14 02:50:46.743745 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:46.743540 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:46.743908 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:46.743758 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hcgk" podUID="153662a6-ddf9-444e-912b-4c6e1beba9d3" Sep 14 02:50:46.743908 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:46.743836 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jrk97" podUID="441362ed-9e44-4875-9e60-ff99bf8fa8b1" Sep 14 02:50:48.745099 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:48.745068 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:48.745737 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:48.745141 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:48.745862 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:48.745174 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:48.745862 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:48.745804 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jrk97" podUID="441362ed-9e44-4875-9e60-ff99bf8fa8b1" Sep 14 02:50:48.745990 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:48.745892 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq8rq" podUID="bc313df4-a60b-4df9-8d21-bb1141d7df64" Sep 14 02:50:48.745990 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:48.745972 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hcgk" podUID="153662a6-ddf9-444e-912b-4c6e1beba9d3" Sep 14 02:50:50.744510 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:50.744190 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:50.745070 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:50.744566 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq8rq" podUID="bc313df4-a60b-4df9-8d21-bb1141d7df64" Sep 14 02:50:50.745070 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:50.744202 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:50.745070 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:50.744190 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:50.745070 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:50.744631 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hcgk" podUID="153662a6-ddf9-444e-912b-4c6e1beba9d3" Sep 14 02:50:50.745070 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:50.744714 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jrk97" podUID="441362ed-9e44-4875-9e60-ff99bf8fa8b1" Sep 14 02:50:50.863889 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:50.863864 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6sn84" event={"ID":"48bf1d45-d3d3-4bde-8f44-688b868762d3","Type":"ContainerStarted","Data":"8e3101c0f8ffb998de714f56ec2645d1f2a0c4e97efa53c6cee5735e525fc11f"} Sep 14 02:50:50.865326 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:50.865305 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tjwgv" event={"ID":"58a26876-794d-43a3-9b28-d8375839bbcf","Type":"ContainerStarted","Data":"f6e13ada663ee883913459a48290190ae78e40533a697cb07fc5aad5d7f26858"} Sep 14 02:50:50.867977 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:50.867144 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xgj42" event={"ID":"06330f4e-f78f-49f6-8c53-9565d0364232","Type":"ContainerStarted","Data":"ee97f8e4e8e69edf2386a38bd039f03aad46160192fae2969ade7d65ffe1df64"} Sep 14 02:50:50.869790 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:50.869766 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-67g56" event={"ID":"4f4515d2-186d-4c05-a47b-afebc4bd9c24","Type":"ContainerStarted","Data":"0ab16ffd4066e04522537bf28b6693fd0140b697f884fd29f47c5a678504624b"} Sep 14 02:50:50.871220 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:50.871198 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n5mdt" event={"ID":"0f5997c3-9c35-4171-aa1b-07f85bb77a17","Type":"ContainerStarted","Data":"1edff069d3101dab7abbad1868f50b96448e71b0ad2785c55f2308532d046802"} Sep 14 02:50:50.872660 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:50.872637 2564 generic.go:356] "Generic (PLEG): container finished" podID="e0c7a5c8-9307-432a-94bc-bdad16bee258" containerID="445a369b5b4823b010f27063ca8ddc36bb738b3c202c4875e5a3bf042ab33ede" exitCode=0 Sep 14 02:50:50.872738 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:50.872717 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjn2n" event={"ID":"e0c7a5c8-9307-432a-94bc-bdad16bee258","Type":"ContainerDied","Data":"445a369b5b4823b010f27063ca8ddc36bb738b3c202c4875e5a3bf042ab33ede"} Sep 14 02:50:50.874411 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:50.874385 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" event={"ID":"c4477e1c-f2aa-4ec6-b278-6027353ca267","Type":"ContainerStarted","Data":"9c34b9f13876f40eaf03928600471d84c6b7b7252bd284b79929751b2b8a7f60"} Sep 14 02:50:50.877315 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:50.877297 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bwv6_4369115a-4032-4f5a-a3e4-c02efe665617/ovn-acl-logging/0.log" Sep 14 02:50:50.877756 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:50.877731 2564 generic.go:356] "Generic (PLEG): container finished" podID="4369115a-4032-4f5a-a3e4-c02efe665617" containerID="1f4fa0d8bd53768b847d5dd04e54cc14ed3cb889f8d15489b89f6182f05d2290" exitCode=1 Sep 14 02:50:50.877831 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:50.877769 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" event={"ID":"4369115a-4032-4f5a-a3e4-c02efe665617","Type":"ContainerStarted","Data":"ab739f1855892a50b1f0eb5d3bdf54fb1ac54a80559e14a5758fdd9189ca2ab8"} Sep 14 02:50:50.877887 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:50.877831 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" event={"ID":"4369115a-4032-4f5a-a3e4-c02efe665617","Type":"ContainerStarted","Data":"a022ff501a7d4c7d39d069b249d1cb26eb5d10bf9662f0f3ccf2ab38ce42d3f7"} Sep 14 02:50:50.877887 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:50.877861 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" event={"ID":"4369115a-4032-4f5a-a3e4-c02efe665617","Type":"ContainerStarted","Data":"e5bea4f714febc799d36e15584f769cd883fd732b7349cd14747f7c9b386f9eb"} Sep 14 02:50:50.878005 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:50.877907 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" event={"ID":"4369115a-4032-4f5a-a3e4-c02efe665617","Type":"ContainerDied","Data":"1f4fa0d8bd53768b847d5dd04e54cc14ed3cb889f8d15489b89f6182f05d2290"} Sep 14 02:50:50.878005 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:50.877956 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" event={"ID":"4369115a-4032-4f5a-a3e4-c02efe665617","Type":"ContainerStarted","Data":"a18ee07c991c0d39d85002f072f85cd929594ce20961349a1f3284bb61368509"} Sep 14 02:50:50.915841 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:50.915793 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6sn84" podStartSLOduration=4.032382744 podStartE2EDuration="21.915773526s" totalImagesPullingTime="17.883390782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:50:50.885441525 +0000 UTC m=+22.615100641" watchObservedRunningTime="2026-09-14 02:50:50.915773526 +0000 UTC m=+22.645432644" Sep 14 02:50:50.970325 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:50.970269 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-67g56" podStartSLOduration=3.999226046 podStartE2EDuration="21.970252762s" totalImagesPullingTime="17.971026716s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:50:50.941042057 +0000 UTC m=+22.670701194" watchObservedRunningTime="2026-09-14 02:50:50.970252762 +0000 UTC m=+22.699911934" Sep 14 02:50:51.002375 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:51.002344 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6sn84" Sep 14 02:50:51.002967 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:51.002940 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6sn84" Sep 14 02:50:51.010855 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:51.010807 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xgj42" podStartSLOduration=4.45931817 podStartE2EDuration="22.010788883s" totalImagesPullingTime="17.551470713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:50:50.970409138 +0000 UTC m=+22.700068261" watchObservedRunningTime="2026-09-14 02:50:51.010788883 +0000 UTC m=+22.740448018" Sep 14 02:50:51.010970 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:51.010895 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tjwgv" podStartSLOduration=4.060317686 podStartE2EDuration="22.010889246s" totalImagesPullingTime="17.95057156s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:50:51.010642464 +0000 UTC m=+22.740301580" watchObservedRunningTime="2026-09-14 02:50:51.010889246 +0000 UTC m=+22.740548376" Sep 14 02:50:51.089276 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:51.089176 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n5mdt" podStartSLOduration=4.073382329 podStartE2EDuration="22.089159244s" totalImagesPullingTime="18.015776915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:50:51.055308403 +0000 UTC m=+22.784967522" watchObservedRunningTime="2026-09-14 02:50:51.089159244 +0000 UTC m=+22.818818364" Sep 14 02:50:51.691349 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:51.691325 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 14 02:50:51.747057 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:51.746897 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-14T02:50:51.691346917Z","UUID":"a7a5e96d-6d0f-44f1-befd-e4d9ac2085d0","Handler":null,"Name":"","Endpoint":""} Sep 14 02:50:51.749923 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:51.749346 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 14 02:50:51.749923 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:51.749383 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 14 02:50:51.881500 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:51.881412 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rxxxt" event={"ID":"50340084-82e7-48f8-93b5-2d99b73389af","Type":"ContainerStarted","Data":"dbdf381edf48c9ebe783a65f44a3a9cf498b3894fb69380018a48626524994ea"} Sep 14 02:50:51.883353 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:51.883323 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" event={"ID":"c4477e1c-f2aa-4ec6-b278-6027353ca267","Type":"ContainerStarted","Data":"fac08b79a02c4559060685c16138cb782a2fd2cdbd8e5c5fb5860062dc6a3362"} Sep 14 02:50:51.887057 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:51.887038 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bwv6_4369115a-4032-4f5a-a3e4-c02efe665617/ovn-acl-logging/0.log" Sep 14 02:50:51.887911 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:51.887505 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" event={"ID":"4369115a-4032-4f5a-a3e4-c02efe665617","Type":"ContainerStarted","Data":"4a68efa10ad16354b36362cd40466e3eec81403921317fd401cb412f06bec3e4"} Sep 14 02:50:51.888028 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:51.887983 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6sn84" Sep 14 02:50:51.888217 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:51.888184 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6sn84" Sep 14 02:50:51.925081 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:51.925027 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rxxxt" podStartSLOduration=4.947952048 podStartE2EDuration="22.925011379s" totalImagesPullingTime="17.977059331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:50:51.901966679 +0000 UTC m=+23.631625797" watchObservedRunningTime="2026-09-14 02:50:51.925011379 +0000 UTC m=+23.654670496" Sep 14 02:50:52.744316 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:52.744288 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:52.744453 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:52.744424 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jrk97" podUID="441362ed-9e44-4875-9e60-ff99bf8fa8b1" Sep 14 02:50:52.744863 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:52.744840 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:52.744955 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:52.744936 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq8rq" podUID="bc313df4-a60b-4df9-8d21-bb1141d7df64" Sep 14 02:50:52.745102 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:52.745083 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:52.745202 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:52.745175 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hcgk" podUID="153662a6-ddf9-444e-912b-4c6e1beba9d3" Sep 14 02:50:52.892163 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:52.892127 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" event={"ID":"c4477e1c-f2aa-4ec6-b278-6027353ca267","Type":"ContainerStarted","Data":"13113aef96d10f9ccefee56c8ac5e818699971504777921eeda333fd4b8d6b9f"} Sep 14 02:50:53.898823 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:53.898791 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bwv6_4369115a-4032-4f5a-a3e4-c02efe665617/ovn-acl-logging/0.log" Sep 14 02:50:53.899289 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:53.899259 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" event={"ID":"4369115a-4032-4f5a-a3e4-c02efe665617","Type":"ContainerStarted","Data":"1e6f673e5cb802560c28bd1cc71a3cc3e602521df0a975761f954bd9c7adac3c"} Sep 14 02:50:54.744389 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:54.744358 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:54.744603 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:54.744358 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:54.744603 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:54.744462 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq8rq" podUID="bc313df4-a60b-4df9-8d21-bb1141d7df64" Sep 14 02:50:54.744603 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:54.744487 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:54.744753 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:54.744617 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hcgk" podUID="153662a6-ddf9-444e-912b-4c6e1beba9d3" Sep 14 02:50:54.744753 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:54.744705 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jrk97" podUID="441362ed-9e44-4875-9e60-ff99bf8fa8b1" Sep 14 02:50:55.905800 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:55.905608 2564 generic.go:356] "Generic (PLEG): container finished" podID="e0c7a5c8-9307-432a-94bc-bdad16bee258" containerID="a159b1377026f41d514bb3e3025257e75bff0d2a4eee93c77b290cc6a3f04fc2" exitCode=0 Sep 14 02:50:55.906540 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:55.905700 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjn2n" event={"ID":"e0c7a5c8-9307-432a-94bc-bdad16bee258","Type":"ContainerDied","Data":"a159b1377026f41d514bb3e3025257e75bff0d2a4eee93c77b290cc6a3f04fc2"} Sep 14 02:50:55.909715 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:55.909696 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bwv6_4369115a-4032-4f5a-a3e4-c02efe665617/ovn-acl-logging/0.log" Sep 14 02:50:55.910110 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:55.910084 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" event={"ID":"4369115a-4032-4f5a-a3e4-c02efe665617","Type":"ContainerStarted","Data":"363d0018608dd44042e137bcd55f1c1ffae609947e28d777cdfaf64a2c767226"} Sep 14 02:50:55.910520 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:55.910503 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:55.910624 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:55.910608 2564 scope.go:117] "RemoveContainer" containerID="1f4fa0d8bd53768b847d5dd04e54cc14ed3cb889f8d15489b89f6182f05d2290" Sep 14 02:50:55.926825 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:55.926793 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:55.933682 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:55.933643 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zbtgx" podStartSLOduration=6.759883962 podStartE2EDuration="26.933628962s" totalImagesPullingTime="20.173745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:50:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:50:52.926471058 +0000 UTC m=+24.656130175" watchObservedRunningTime="2026-09-14 02:50:55.933628962 +0000 UTC m=+27.663288079" Sep 14 02:50:56.596491 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:56.596446 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:56.745143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:56.744604 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:56.745143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:56.744646 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:56.745143 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:56.744758 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jrk97" podUID="441362ed-9e44-4875-9e60-ff99bf8fa8b1" Sep 14 02:50:56.745143 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:56.744806 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq8rq" podUID="bc313df4-a60b-4df9-8d21-bb1141d7df64" Sep 14 02:50:56.745143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:56.744862 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:56.745143 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:56.744916 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hcgk" podUID="153662a6-ddf9-444e-912b-4c6e1beba9d3" Sep 14 02:50:56.915963 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:56.915809 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bwv6_4369115a-4032-4f5a-a3e4-c02efe665617/ovn-acl-logging/0.log" Sep 14 02:50:56.916313 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:56.916291 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" event={"ID":"4369115a-4032-4f5a-a3e4-c02efe665617","Type":"ContainerStarted","Data":"888eb0cf892972a2f203ca4f09c65a82b03506d7cb217e3e853f25d6a85bcea5"} Sep 14 02:50:56.916631 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:56.916602 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:56.932258 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:56.932243 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:50:56.970302 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:56.970252 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" podStartSLOduration=9.957249885 podStartE2EDuration="27.970233834s" totalImagesPullingTime="18.012983949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:50:56.970038107 +0000 UTC m=+28.699697225" watchObservedRunningTime="2026-09-14 02:50:56.970233834 +0000 UTC m=+28.699892989" Sep 14 02:50:57.630058 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:57.629978 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jrk97"] Sep 14 02:50:57.630216 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:57.630078 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:57.630216 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:57.630162 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jrk97" podUID="441362ed-9e44-4875-9e60-ff99bf8fa8b1" Sep 14 02:50:57.635831 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:57.635809 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5hcgk"] Sep 14 02:50:57.635939 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:57.635876 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:57.635973 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:57.635939 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hcgk" podUID="153662a6-ddf9-444e-912b-4c6e1beba9d3" Sep 14 02:50:57.650966 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:57.650945 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mq8rq"] Sep 14 02:50:57.651055 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:57.651017 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:57.651118 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:57.651088 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq8rq" podUID="bc313df4-a60b-4df9-8d21-bb1141d7df64" Sep 14 02:50:57.920667 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:57.920633 2564 generic.go:356] "Generic (PLEG): container finished" podID="e0c7a5c8-9307-432a-94bc-bdad16bee258" containerID="d10a9aba49f85550569957b86638eaf6004510097b5c7515bd9ea5ecbc2ebd1f" exitCode=0 Sep 14 02:50:57.921054 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:57.920711 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjn2n" event={"ID":"e0c7a5c8-9307-432a-94bc-bdad16bee258","Type":"ContainerDied","Data":"d10a9aba49f85550569957b86638eaf6004510097b5c7515bd9ea5ecbc2ebd1f"} Sep 14 02:50:58.924651 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:58.924591 2564 generic.go:356] "Generic (PLEG): container finished" podID="e0c7a5c8-9307-432a-94bc-bdad16bee258" containerID="e24bc20b3aed62d67789956b5dc2910a606251e5e2643fb7fcfe68e80c6272c7" exitCode=0 Sep 14 02:50:58.924968 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:58.924672 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjn2n" event={"ID":"e0c7a5c8-9307-432a-94bc-bdad16bee258","Type":"ContainerDied","Data":"e24bc20b3aed62d67789956b5dc2910a606251e5e2643fb7fcfe68e80c6272c7"} Sep 14 02:50:59.743724 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:59.743652 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:50:59.743870 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:59.743763 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:50:59.743870 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:59.743771 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq8rq" podUID="bc313df4-a60b-4df9-8d21-bb1141d7df64" Sep 14 02:50:59.743870 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:59.743849 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hcgk" podUID="153662a6-ddf9-444e-912b-4c6e1beba9d3" Sep 14 02:50:59.744030 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:50:59.743910 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:50:59.744074 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:50:59.744028 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jrk97" podUID="441362ed-9e44-4875-9e60-ff99bf8fa8b1" Sep 14 02:51:01.743701 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:01.743675 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:51:01.743701 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:01.743700 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:51:01.744238 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:01.743681 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:51:01.744238 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:01.743775 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5hcgk" podUID="153662a6-ddf9-444e-912b-4c6e1beba9d3" Sep 14 02:51:01.744238 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:01.743854 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mq8rq" podUID="bc313df4-a60b-4df9-8d21-bb1141d7df64" Sep 14 02:51:01.744238 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:01.743924 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jrk97" podUID="441362ed-9e44-4875-9e60-ff99bf8fa8b1" Sep 14 02:51:02.540546 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:02.540501 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret\") pod \"global-pull-secret-syncer-mq8rq\" (UID: \"bc313df4-a60b-4df9-8d21-bb1141d7df64\") " pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:51:02.540825 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:02.540742 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs\") pod \"network-metrics-daemon-jrk97\" (UID: \"441362ed-9e44-4875-9e60-ff99bf8fa8b1\") " pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:51:02.540825 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:02.540664 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 02:51:02.540926 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:02.540844 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret podName:bc313df4-a60b-4df9-8d21-bb1141d7df64 nodeName:}" failed. No retries permitted until 2026-09-14 02:51:34.540818777 +0000 UTC m=+66.270477885 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret") pod "global-pull-secret-syncer-mq8rq" (UID: "bc313df4-a60b-4df9-8d21-bb1141d7df64") : object "kube-system"/"original-pull-secret" not registered Sep 14 02:51:02.540926 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:02.540845 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 02:51:02.540926 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:02.540900 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs podName:441362ed-9e44-4875-9e60-ff99bf8fa8b1 nodeName:}" failed. No retries permitted until 2026-09-14 02:51:34.540883388 +0000 UTC m=+66.270542501 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs") pod "network-metrics-daemon-jrk97" (UID: "441362ed-9e44-4875-9e60-ff99bf8fa8b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 02:51:02.641650 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:02.641611 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f27m4\" (UniqueName: \"kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4\") pod \"network-check-target-5hcgk\" (UID: \"153662a6-ddf9-444e-912b-4c6e1beba9d3\") " pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:51:02.641847 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:02.641779 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 02:51:02.641847 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:02.641796 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 02:51:02.641847 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:02.641806 2564 projected.go:196] Error preparing data for projected volume kube-api-access-f27m4 for pod openshift-network-diagnostics/network-check-target-5hcgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 02:51:02.641976 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:02.641860 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4 podName:153662a6-ddf9-444e-912b-4c6e1beba9d3 nodeName:}" failed. No retries permitted until 2026-09-14 02:51:34.641845781 +0000 UTC m=+66.371504894 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-f27m4" (UniqueName: "kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4") pod "network-check-target-5hcgk" (UID: "153662a6-ddf9-444e-912b-4c6e1beba9d3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 02:51:03.075942 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.075860 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-88.ec2.internal" event="NodeReady" Sep 14 02:51:03.076376 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.076016 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 14 02:51:03.139816 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.139775 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76c79f4bcc-mg4xz"] Sep 14 02:51:03.193056 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.193024 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76c79f4bcc-mg4xz"] Sep 14 02:51:03.193201 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.193167 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.198817 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.198788 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 14 02:51:03.200723 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.200701 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 14 02:51:03.201004 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.200987 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tz6jr\"" Sep 14 02:51:03.202106 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.201228 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 14 02:51:03.210680 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.210662 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 14 02:51:03.250753 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.250723 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-znkd5"] Sep 14 02:51:03.272106 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.272085 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8px75"] Sep 14 02:51:03.272224 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.272206 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-znkd5" Sep 14 02:51:03.276941 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.276917 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 14 02:51:03.277058 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.277041 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 14 02:51:03.277142 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.277129 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nzbvq\"" Sep 14 02:51:03.296488 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.296446 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-znkd5"] Sep 14 02:51:03.296607 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.296579 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8px75" Sep 14 02:51:03.300087 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.300067 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 14 02:51:03.300382 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.300366 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 14 02:51:03.301130 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.301112 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-hnrbf\"" Sep 14 02:51:03.301228 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.301119 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 14 02:51:03.322683 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.322657 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8px75"] Sep 14 02:51:03.346773 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.346704 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sczvk\" (UniqueName: \"kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-kube-api-access-sczvk\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.346773 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.346743 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.346773 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.346772 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba5bd25d-67d4-45cf-a2fa-e5169274172b-ca-trust-extracted\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.347002 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.346849 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba5bd25d-67d4-45cf-a2fa-e5169274172b-trusted-ca\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.347002 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.346886 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-bound-sa-token\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.347002 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.346925 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/47b16be3-755a-4a89-a2ef-830c72498f05-config-volume\") pod \"dns-default-znkd5\" (UID: \"47b16be3-755a-4a89-a2ef-830c72498f05\") " pod="openshift-dns/dns-default-znkd5" Sep 14 02:51:03.347002 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.346953 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cfcl\" (UniqueName: \"kubernetes.io/projected/47b16be3-755a-4a89-a2ef-830c72498f05-kube-api-access-5cfcl\") pod \"dns-default-znkd5\" (UID: \"47b16be3-755a-4a89-a2ef-830c72498f05\") " pod="openshift-dns/dns-default-znkd5" Sep 14 02:51:03.347002 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.346979 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba5bd25d-67d4-45cf-a2fa-e5169274172b-image-registry-private-configuration\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.347002 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.347003 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47b16be3-755a-4a89-a2ef-830c72498f05-tmp-dir\") pod \"dns-default-znkd5\" (UID: \"47b16be3-755a-4a89-a2ef-830c72498f05\") " pod="openshift-dns/dns-default-znkd5" Sep 14 02:51:03.347291 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.347021 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-certificates\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.347291 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.347038 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba5bd25d-67d4-45cf-a2fa-e5169274172b-installation-pull-secrets\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.347291 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.347073 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls\") pod \"dns-default-znkd5\" (UID: \"47b16be3-755a-4a89-a2ef-830c72498f05\") " pod="openshift-dns/dns-default-znkd5" Sep 14 02:51:03.448517 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.448316 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-certificates\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.448682 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.448530 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba5bd25d-67d4-45cf-a2fa-e5169274172b-installation-pull-secrets\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.448682 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.448553 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls\") pod \"dns-default-znkd5\" (UID: \"47b16be3-755a-4a89-a2ef-830c72498f05\") " pod="openshift-dns/dns-default-znkd5" Sep 14 02:51:03.448682 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.448607 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b6e2fea-3457-4b62-ba40-0254ed00fb1e-cert\") pod \"ingress-canary-8px75\" (UID: \"1b6e2fea-3457-4b62-ba40-0254ed00fb1e\") " pod="openshift-ingress-canary/ingress-canary-8px75" Sep 14 02:51:03.448682 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.448636 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sczvk\" (UniqueName: \"kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-kube-api-access-sczvk\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.448682 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.448664 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.448920 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.448686 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba5bd25d-67d4-45cf-a2fa-e5169274172b-ca-trust-extracted\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.448920 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:03.448701 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 02:51:03.448920 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.448714 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba5bd25d-67d4-45cf-a2fa-e5169274172b-trusted-ca\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.448920 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.448735 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-bound-sa-token\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.448920 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:03.448760 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls podName:47b16be3-755a-4a89-a2ef-830c72498f05 nodeName:}" failed. No retries permitted until 2026-09-14 02:51:03.948745188 +0000 UTC m=+35.678404283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls") pod "dns-default-znkd5" (UID: "47b16be3-755a-4a89-a2ef-830c72498f05") : secret "dns-default-metrics-tls" not found Sep 14 02:51:03.448920 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.448811 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/47b16be3-755a-4a89-a2ef-830c72498f05-config-volume\") pod \"dns-default-znkd5\" (UID: \"47b16be3-755a-4a89-a2ef-830c72498f05\") " pod="openshift-dns/dns-default-znkd5" Sep 14 02:51:03.448920 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.448843 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k45qm\" (UniqueName: \"kubernetes.io/projected/1b6e2fea-3457-4b62-ba40-0254ed00fb1e-kube-api-access-k45qm\") pod \"ingress-canary-8px75\" (UID: \"1b6e2fea-3457-4b62-ba40-0254ed00fb1e\") " pod="openshift-ingress-canary/ingress-canary-8px75" Sep 14 02:51:03.448920 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.448873 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cfcl\" (UniqueName: \"kubernetes.io/projected/47b16be3-755a-4a89-a2ef-830c72498f05-kube-api-access-5cfcl\") pod \"dns-default-znkd5\" (UID: \"47b16be3-755a-4a89-a2ef-830c72498f05\") " pod="openshift-dns/dns-default-znkd5" Sep 14 02:51:03.448920 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.448905 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba5bd25d-67d4-45cf-a2fa-e5169274172b-image-registry-private-configuration\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.449346 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.448940 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47b16be3-755a-4a89-a2ef-830c72498f05-tmp-dir\") pod \"dns-default-znkd5\" (UID: \"47b16be3-755a-4a89-a2ef-830c72498f05\") " pod="openshift-dns/dns-default-znkd5" Sep 14 02:51:03.449346 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:03.449050 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 02:51:03.449346 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:03.449062 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c79f4bcc-mg4xz: secret "image-registry-tls" not found Sep 14 02:51:03.449346 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:03.449107 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls podName:ba5bd25d-67d4-45cf-a2fa-e5169274172b nodeName:}" failed. No retries permitted until 2026-09-14 02:51:03.949090573 +0000 UTC m=+35.678749668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls") pod "image-registry-76c79f4bcc-mg4xz" (UID: "ba5bd25d-67d4-45cf-a2fa-e5169274172b") : secret "image-registry-tls" not found Sep 14 02:51:03.449346 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.449246 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47b16be3-755a-4a89-a2ef-830c72498f05-tmp-dir\") pod \"dns-default-znkd5\" (UID: \"47b16be3-755a-4a89-a2ef-830c72498f05\") " pod="openshift-dns/dns-default-znkd5" Sep 14 02:51:03.449346 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.449261 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba5bd25d-67d4-45cf-a2fa-e5169274172b-ca-trust-extracted\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.449346 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.449315 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-certificates\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.449812 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.449790 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/47b16be3-755a-4a89-a2ef-830c72498f05-config-volume\") pod \"dns-default-znkd5\" (UID: \"47b16be3-755a-4a89-a2ef-830c72498f05\") " pod="openshift-dns/dns-default-znkd5" Sep 14 02:51:03.449923 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.449901 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba5bd25d-67d4-45cf-a2fa-e5169274172b-trusted-ca\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.453418 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.453391 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba5bd25d-67d4-45cf-a2fa-e5169274172b-installation-pull-secrets\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.453560 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.453462 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba5bd25d-67d4-45cf-a2fa-e5169274172b-image-registry-private-configuration\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.460657 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.460636 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-bound-sa-token\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.463184 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.463160 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cfcl\" (UniqueName: \"kubernetes.io/projected/47b16be3-755a-4a89-a2ef-830c72498f05-kube-api-access-5cfcl\") pod \"dns-default-znkd5\" (UID: \"47b16be3-755a-4a89-a2ef-830c72498f05\") " pod="openshift-dns/dns-default-znkd5" Sep 14 02:51:03.464019 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.464001 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sczvk\" (UniqueName: \"kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-kube-api-access-sczvk\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.549752 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.549729 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k45qm\" (UniqueName: \"kubernetes.io/projected/1b6e2fea-3457-4b62-ba40-0254ed00fb1e-kube-api-access-k45qm\") pod \"ingress-canary-8px75\" (UID: \"1b6e2fea-3457-4b62-ba40-0254ed00fb1e\") " pod="openshift-ingress-canary/ingress-canary-8px75" Sep 14 02:51:03.549869 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.549792 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b6e2fea-3457-4b62-ba40-0254ed00fb1e-cert\") pod \"ingress-canary-8px75\" (UID: \"1b6e2fea-3457-4b62-ba40-0254ed00fb1e\") " pod="openshift-ingress-canary/ingress-canary-8px75" Sep 14 02:51:03.552097 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.552072 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1b6e2fea-3457-4b62-ba40-0254ed00fb1e-cert\") pod \"ingress-canary-8px75\" (UID: \"1b6e2fea-3457-4b62-ba40-0254ed00fb1e\") " pod="openshift-ingress-canary/ingress-canary-8px75" Sep 14 02:51:03.566135 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.566111 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k45qm\" (UniqueName: \"kubernetes.io/projected/1b6e2fea-3457-4b62-ba40-0254ed00fb1e-kube-api-access-k45qm\") pod \"ingress-canary-8px75\" (UID: \"1b6e2fea-3457-4b62-ba40-0254ed00fb1e\") " pod="openshift-ingress-canary/ingress-canary-8px75" Sep 14 02:51:03.606416 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.606368 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8px75" Sep 14 02:51:03.743903 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.743876 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:51:03.744050 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.743922 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:51:03.744050 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.743935 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:51:03.754517 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.754493 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 14 02:51:03.754655 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.754638 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b88wl\"" Sep 14 02:51:03.754760 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.754742 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 14 02:51:03.755530 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.755510 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 14 02:51:03.755708 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.755693 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mk9cr\"" Sep 14 02:51:03.755762 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.755716 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 14 02:51:03.953383 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.953342 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:03.953528 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.953439 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls\") pod \"dns-default-znkd5\" (UID: \"47b16be3-755a-4a89-a2ef-830c72498f05\") " pod="openshift-dns/dns-default-znkd5" Sep 14 02:51:03.953600 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:03.953568 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 02:51:03.953600 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:03.953587 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c79f4bcc-mg4xz: secret "image-registry-tls" not found Sep 14 02:51:03.953600 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:03.953592 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 02:51:03.953740 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:03.953648 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls podName:ba5bd25d-67d4-45cf-a2fa-e5169274172b nodeName:}" failed. No retries permitted until 2026-09-14 02:51:04.953627717 +0000 UTC m=+36.683286835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls") pod "image-registry-76c79f4bcc-mg4xz" (UID: "ba5bd25d-67d4-45cf-a2fa-e5169274172b") : secret "image-registry-tls" not found Sep 14 02:51:03.953740 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:03.953664 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls podName:47b16be3-755a-4a89-a2ef-830c72498f05 nodeName:}" failed. No retries permitted until 2026-09-14 02:51:04.953657963 +0000 UTC m=+36.683317058 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls") pod "dns-default-znkd5" (UID: "47b16be3-755a-4a89-a2ef-830c72498f05") : secret "dns-default-metrics-tls" not found Sep 14 02:51:03.956960 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:03.956934 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8px75"] Sep 14 02:51:03.958539 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:51:03.958515 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6e2fea_3457_4b62_ba40_0254ed00fb1e.slice/crio-39b8989d277cf0768ae510a370e71202d9157cb61475a237e60eaa8c6556b434 WatchSource:0}: Error finding container 39b8989d277cf0768ae510a370e71202d9157cb61475a237e60eaa8c6556b434: Status 404 returned error can't find the container with id 39b8989d277cf0768ae510a370e71202d9157cb61475a237e60eaa8c6556b434 Sep 14 02:51:04.942076 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:04.942039 2564 generic.go:356] "Generic (PLEG): container finished" podID="e0c7a5c8-9307-432a-94bc-bdad16bee258" containerID="b178a14fb55e0363468560e5719604a9c54af66a66cb4fce50e43d31ae2958b2" exitCode=0 Sep 14 02:51:04.942973 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:04.942126 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjn2n" event={"ID":"e0c7a5c8-9307-432a-94bc-bdad16bee258","Type":"ContainerDied","Data":"b178a14fb55e0363468560e5719604a9c54af66a66cb4fce50e43d31ae2958b2"} Sep 14 02:51:04.943281 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:04.943166 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8px75" event={"ID":"1b6e2fea-3457-4b62-ba40-0254ed00fb1e","Type":"ContainerStarted","Data":"39b8989d277cf0768ae510a370e71202d9157cb61475a237e60eaa8c6556b434"} Sep 14 02:51:04.961592 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:04.961561 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls\") pod \"dns-default-znkd5\" (UID: \"47b16be3-755a-4a89-a2ef-830c72498f05\") " pod="openshift-dns/dns-default-znkd5" Sep 14 02:51:04.961710 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:04.961638 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:04.961768 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:04.961740 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 02:51:04.961823 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:04.961773 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 02:51:04.961823 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:04.961787 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c79f4bcc-mg4xz: secret "image-registry-tls" not found Sep 14 02:51:04.961823 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:04.961804 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls podName:47b16be3-755a-4a89-a2ef-830c72498f05 nodeName:}" failed. No retries permitted until 2026-09-14 02:51:06.96178443 +0000 UTC m=+38.691443542 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls") pod "dns-default-znkd5" (UID: "47b16be3-755a-4a89-a2ef-830c72498f05") : secret "dns-default-metrics-tls" not found Sep 14 02:51:04.961972 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:04.961828 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls podName:ba5bd25d-67d4-45cf-a2fa-e5169274172b nodeName:}" failed. No retries permitted until 2026-09-14 02:51:06.961817505 +0000 UTC m=+38.691476599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls") pod "image-registry-76c79f4bcc-mg4xz" (UID: "ba5bd25d-67d4-45cf-a2fa-e5169274172b") : secret "image-registry-tls" not found Sep 14 02:51:05.950210 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:05.949978 2564 generic.go:356] "Generic (PLEG): container finished" podID="e0c7a5c8-9307-432a-94bc-bdad16bee258" containerID="144608cd4ebccbfc9cc3fab52b260f006f81eb08b9fd11c9033918803a2cecc6" exitCode=0 Sep 14 02:51:05.950210 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:05.950079 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjn2n" event={"ID":"e0c7a5c8-9307-432a-94bc-bdad16bee258","Type":"ContainerDied","Data":"144608cd4ebccbfc9cc3fab52b260f006f81eb08b9fd11c9033918803a2cecc6"} Sep 14 02:51:06.955913 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:06.955881 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fjn2n" event={"ID":"e0c7a5c8-9307-432a-94bc-bdad16bee258","Type":"ContainerStarted","Data":"34a09500c4e6dcd3b6634448fbc8557ea098e16752a5cbddb2095efa34fb7f1f"} Sep 14 02:51:06.957399 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:06.957376 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8px75" event={"ID":"1b6e2fea-3457-4b62-ba40-0254ed00fb1e","Type":"ContainerStarted","Data":"f4a1ba8ac9ade2912af3800bae1c1e46af250c1061bb81e10ba90da9619ba0c6"} Sep 14 02:51:06.981383 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:06.978834 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:06.981383 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:06.978961 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 02:51:06.981383 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:06.978976 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c79f4bcc-mg4xz: secret "image-registry-tls" not found Sep 14 02:51:06.981383 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:06.978995 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls\") pod \"dns-default-znkd5\" (UID: \"47b16be3-755a-4a89-a2ef-830c72498f05\") " pod="openshift-dns/dns-default-znkd5" Sep 14 02:51:06.981383 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:06.979050 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls podName:ba5bd25d-67d4-45cf-a2fa-e5169274172b nodeName:}" failed. No retries permitted until 2026-09-14 02:51:10.979021446 +0000 UTC m=+42.708680543 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls") pod "image-registry-76c79f4bcc-mg4xz" (UID: "ba5bd25d-67d4-45cf-a2fa-e5169274172b") : secret "image-registry-tls" not found Sep 14 02:51:06.981383 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:06.979113 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 02:51:06.981383 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:06.979169 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls podName:47b16be3-755a-4a89-a2ef-830c72498f05 nodeName:}" failed. No retries permitted until 2026-09-14 02:51:10.979152278 +0000 UTC m=+42.708811388 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls") pod "dns-default-znkd5" (UID: "47b16be3-755a-4a89-a2ef-830c72498f05") : secret "dns-default-metrics-tls" not found Sep 14 02:51:06.983116 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:06.983080 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fjn2n" podStartSLOduration=8.71146281 podStartE2EDuration="37.983067508s" totalImagesPullingTime="29.271604698s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:50:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:51:06.982059935 +0000 UTC m=+38.711719052" watchObservedRunningTime="2026-09-14 02:51:06.983067508 +0000 UTC m=+38.712726624" Sep 14 02:51:07.003238 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:07.003147 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8px75" podStartSLOduration=1.324306331 podStartE2EDuration="4.003131704s" totalImagesPullingTime="2.678825373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:51:07.002361881 +0000 UTC m=+38.732020998" watchObservedRunningTime="2026-09-14 02:51:07.003131704 +0000 UTC m=+38.732790833" Sep 14 02:51:07.179565 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:07.179528 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:07.345929 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:07.345829 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:08.202127 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:08.202085 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:08.302375 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:08.302346 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:11.008177 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:11.008118 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:11.008571 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:11.008219 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls\") pod \"dns-default-znkd5\" (UID: \"47b16be3-755a-4a89-a2ef-830c72498f05\") " pod="openshift-dns/dns-default-znkd5" Sep 14 02:51:11.008571 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:11.008268 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 02:51:11.008571 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:11.008285 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c79f4bcc-mg4xz: secret "image-registry-tls" not found Sep 14 02:51:11.008571 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:11.008329 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 02:51:11.008571 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:11.008336 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls podName:ba5bd25d-67d4-45cf-a2fa-e5169274172b nodeName:}" failed. No retries permitted until 2026-09-14 02:51:19.008321954 +0000 UTC m=+50.737981049 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls") pod "image-registry-76c79f4bcc-mg4xz" (UID: "ba5bd25d-67d4-45cf-a2fa-e5169274172b") : secret "image-registry-tls" not found Sep 14 02:51:11.008571 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:11.008369 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls podName:47b16be3-755a-4a89-a2ef-830c72498f05 nodeName:}" failed. No retries permitted until 2026-09-14 02:51:19.008358462 +0000 UTC m=+50.738017556 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls") pod "dns-default-znkd5" (UID: "47b16be3-755a-4a89-a2ef-830c72498f05") : secret "dns-default-metrics-tls" not found Sep 14 02:51:14.161943 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:14.161903 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:14.290772 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:14.290738 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:15.023322 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:15.023275 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:15.243332 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:15.243303 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:15.345604 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:15.345533 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:15.460456 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:15.460421 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:15.731653 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:15.731618 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:15.827743 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:15.827702 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:15.945066 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:15.945036 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:16.057595 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:16.057496 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:16.757652 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:16.757610 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:16.871937 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:16.871887 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:18.598289 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:18.598252 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:19.068908 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:19.068873 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls\") pod \"dns-default-znkd5\" (UID: \"47b16be3-755a-4a89-a2ef-830c72498f05\") " pod="openshift-dns/dns-default-znkd5" Sep 14 02:51:19.069072 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:19.068929 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:19.069072 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:19.069017 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 02:51:19.069072 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:19.069046 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 02:51:19.069072 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:19.069057 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c79f4bcc-mg4xz: secret "image-registry-tls" not found Sep 14 02:51:19.069197 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:19.069085 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls podName:47b16be3-755a-4a89-a2ef-830c72498f05 nodeName:}" failed. No retries permitted until 2026-09-14 02:51:35.069069702 +0000 UTC m=+66.798728797 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls") pod "dns-default-znkd5" (UID: "47b16be3-755a-4a89-a2ef-830c72498f05") : secret "dns-default-metrics-tls" not found Sep 14 02:51:19.069197 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:19.069102 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls podName:ba5bd25d-67d4-45cf-a2fa-e5169274172b nodeName:}" failed. No retries permitted until 2026-09-14 02:51:35.069090954 +0000 UTC m=+66.798750049 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls") pod "image-registry-76c79f4bcc-mg4xz" (UID: "ba5bd25d-67d4-45cf-a2fa-e5169274172b") : secret "image-registry-tls" not found Sep 14 02:51:20.123518 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:20.123465 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:20.623007 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:20.622975 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:28.935437 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:28.935407 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5bwv6" Sep 14 02:51:29.172023 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:29.171984 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:29.273671 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:29.273590 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:29.388121 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:29.388091 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:29.472371 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:29.472340 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:29.571836 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:29.571761 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:31.356930 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:31.356886 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:31.469966 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:31.469930 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:32.630472 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:32.630437 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:32.741881 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:32.741842 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:32.893657 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:32.893626 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:33.029672 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:33.029647 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:33.269147 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:33.269057 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:33.374668 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:33.374628 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:34.581492 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:34.581439 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret\") pod \"global-pull-secret-syncer-mq8rq\" (UID: \"bc313df4-a60b-4df9-8d21-bb1141d7df64\") " pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:51:34.581882 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:34.581521 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs\") pod \"network-metrics-daemon-jrk97\" (UID: \"441362ed-9e44-4875-9e60-ff99bf8fa8b1\") " pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:51:34.584203 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:34.584183 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 14 02:51:34.584253 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:34.584185 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 14 02:51:34.592114 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:34.592091 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 14 02:51:34.592197 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:34.592175 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs podName:441362ed-9e44-4875-9e60-ff99bf8fa8b1 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:38.592153121 +0000 UTC m=+130.321812216 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs") pod "network-metrics-daemon-jrk97" (UID: "441362ed-9e44-4875-9e60-ff99bf8fa8b1") : secret "metrics-daemon-secret" not found Sep 14 02:51:34.595456 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:34.595434 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc313df4-a60b-4df9-8d21-bb1141d7df64-original-pull-secret\") pod \"global-pull-secret-syncer-mq8rq\" (UID: \"bc313df4-a60b-4df9-8d21-bb1141d7df64\") " pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:51:34.654239 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:34.654211 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mq8rq" Sep 14 02:51:34.682702 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:34.682682 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f27m4\" (UniqueName: \"kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4\") pod \"network-check-target-5hcgk\" (UID: \"153662a6-ddf9-444e-912b-4c6e1beba9d3\") " pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:51:34.686092 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:34.686073 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 14 02:51:34.696362 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:34.696335 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 14 02:51:34.705981 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:34.705952 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f27m4\" (UniqueName: \"kubernetes.io/projected/153662a6-ddf9-444e-912b-4c6e1beba9d3-kube-api-access-f27m4\") pod \"network-check-target-5hcgk\" (UID: \"153662a6-ddf9-444e-912b-4c6e1beba9d3\") " pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:51:34.954142 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:34.954099 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mq8rq"] Sep 14 02:51:34.957412 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:51:34.957387 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc313df4_a60b_4df9_8d21_bb1141d7df64.slice/crio-34b76bd2e4ead065f16f103fbc76567f75b224038d9497c5e9ef77ad61ec2464 WatchSource:0}: Error finding container 34b76bd2e4ead065f16f103fbc76567f75b224038d9497c5e9ef77ad61ec2464: Status 404 returned error can't find the container with id 34b76bd2e4ead065f16f103fbc76567f75b224038d9497c5e9ef77ad61ec2464 Sep 14 02:51:34.966624 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:34.966604 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b88wl\"" Sep 14 02:51:34.969664 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:34.969645 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:51:35.027327 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:35.027292 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mq8rq" event={"ID":"bc313df4-a60b-4df9-8d21-bb1141d7df64","Type":"ContainerStarted","Data":"34b76bd2e4ead065f16f103fbc76567f75b224038d9497c5e9ef77ad61ec2464"} Sep 14 02:51:35.087382 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:35.087346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:51:35.087570 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:35.087419 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls\") pod \"dns-default-znkd5\" (UID: \"47b16be3-755a-4a89-a2ef-830c72498f05\") " pod="openshift-dns/dns-default-znkd5" Sep 14 02:51:35.087570 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:35.087522 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 02:51:35.087570 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:35.087535 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 02:51:35.087570 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:35.087543 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c79f4bcc-mg4xz: secret "image-registry-tls" not found Sep 14 02:51:35.087707 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:35.087605 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls podName:ba5bd25d-67d4-45cf-a2fa-e5169274172b nodeName:}" failed. No retries permitted until 2026-09-14 02:52:07.087585292 +0000 UTC m=+98.817244408 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls") pod "image-registry-76c79f4bcc-mg4xz" (UID: "ba5bd25d-67d4-45cf-a2fa-e5169274172b") : secret "image-registry-tls" not found Sep 14 02:51:35.087707 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:51:35.087619 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls podName:47b16be3-755a-4a89-a2ef-830c72498f05 nodeName:}" failed. No retries permitted until 2026-09-14 02:52:07.087613184 +0000 UTC m=+98.817272279 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls") pod "dns-default-znkd5" (UID: "47b16be3-755a-4a89-a2ef-830c72498f05") : secret "dns-default-metrics-tls" not found Sep 14 02:51:35.267981 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:35.267912 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5hcgk"] Sep 14 02:51:35.270880 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:51:35.270851 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153662a6_ddf9_444e_912b_4c6e1beba9d3.slice/crio-ca4e0dba116921502603763cf191fe0c1f12d7664212bfff30e43f050b9d1348 WatchSource:0}: Error finding container ca4e0dba116921502603763cf191fe0c1f12d7664212bfff30e43f050b9d1348: Status 404 returned error can't find the container with id ca4e0dba116921502603763cf191fe0c1f12d7664212bfff30e43f050b9d1348 Sep 14 02:51:36.031416 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:36.031344 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5hcgk" event={"ID":"153662a6-ddf9-444e-912b-4c6e1beba9d3","Type":"ContainerStarted","Data":"ca4e0dba116921502603763cf191fe0c1f12d7664212bfff30e43f050b9d1348"} Sep 14 02:51:40.048769 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:40.048732 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5hcgk" event={"ID":"153662a6-ddf9-444e-912b-4c6e1beba9d3","Type":"ContainerStarted","Data":"c1704a879340c31c927fd00df9dc258cad6c8c2c3faa860b435d1a9957ed263e"} Sep 14 02:51:40.049161 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:40.048921 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:51:40.050412 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:40.050388 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mq8rq" event={"ID":"bc313df4-a60b-4df9-8d21-bb1141d7df64","Type":"ContainerStarted","Data":"4040c716bbac7a622383e536b7be8d9c09018c8852c7899349cdbd81302f41c9"} Sep 14 02:51:40.065418 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:40.065378 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5hcgk" podStartSLOduration=66.408172625 podStartE2EDuration="1m11.06536331s" totalImagesPullingTime="4.657190685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:51:40.064720643 +0000 UTC m=+71.794379763" watchObservedRunningTime="2026-09-14 02:51:40.06536331 +0000 UTC m=+71.795022426" Sep 14 02:51:40.080874 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:40.080769 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mq8rq" podStartSLOduration=66.1086881 podStartE2EDuration="1m11.080757037s" totalImagesPullingTime="4.972068937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:51:40.080660776 +0000 UTC m=+71.810319893" watchObservedRunningTime="2026-09-14 02:51:40.080757037 +0000 UTC m=+71.810416155" Sep 14 02:51:53.111350 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:53.111318 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:53.172025 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:53.171997 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:53.739436 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:53.739395 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:53.837394 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:53.837353 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:59.053718 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:59.053683 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:51:59.154592 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:51:59.154552 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:01.829927 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:01.829892 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:01.900608 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:01.900578 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:03.889949 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:03.889914 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:03.956422 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:03.956381 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:04.393433 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:04.393390 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:04.512992 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:04.512953 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:04.662968 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:04.662891 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:04.756945 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:04.756906 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:06.226701 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:06.226663 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-dkg2s"] Sep 14 02:52:06.233174 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:06.233139 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-dkg2s" Sep 14 02:52:06.237759 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:06.237715 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 14 02:52:06.237759 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:06.237736 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 14 02:52:06.238073 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:06.237721 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 14 02:52:06.238594 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:06.238573 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-28jq6\"" Sep 14 02:52:06.238714 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:06.238636 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 14 02:52:06.239770 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:06.239752 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-dkg2s"] Sep 14 02:52:06.282427 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:06.282391 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:06.306318 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:06.306292 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6e86c36d-22a6-4e70-8664-b7fce6d95869-signing-key\") pod \"service-ca-568869fdc5-dkg2s\" (UID: \"6e86c36d-22a6-4e70-8664-b7fce6d95869\") " pod="openshift-service-ca/service-ca-568869fdc5-dkg2s" Sep 14 02:52:06.306419 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:06.306348 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6e86c36d-22a6-4e70-8664-b7fce6d95869-signing-cabundle\") pod \"service-ca-568869fdc5-dkg2s\" (UID: \"6e86c36d-22a6-4e70-8664-b7fce6d95869\") " pod="openshift-service-ca/service-ca-568869fdc5-dkg2s" Sep 14 02:52:06.306419 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:06.306368 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gc9d\" (UniqueName: \"kubernetes.io/projected/6e86c36d-22a6-4e70-8664-b7fce6d95869-kube-api-access-7gc9d\") pod \"service-ca-568869fdc5-dkg2s\" (UID: \"6e86c36d-22a6-4e70-8664-b7fce6d95869\") " pod="openshift-service-ca/service-ca-568869fdc5-dkg2s" Sep 14 02:52:06.407678 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:06.407645 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6e86c36d-22a6-4e70-8664-b7fce6d95869-signing-cabundle\") pod \"service-ca-568869fdc5-dkg2s\" (UID: \"6e86c36d-22a6-4e70-8664-b7fce6d95869\") " pod="openshift-service-ca/service-ca-568869fdc5-dkg2s" Sep 14 02:52:06.407678 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:06.407683 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gc9d\" (UniqueName: \"kubernetes.io/projected/6e86c36d-22a6-4e70-8664-b7fce6d95869-kube-api-access-7gc9d\") pod \"service-ca-568869fdc5-dkg2s\" (UID: \"6e86c36d-22a6-4e70-8664-b7fce6d95869\") " pod="openshift-service-ca/service-ca-568869fdc5-dkg2s" Sep 14 02:52:06.407853 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:06.407757 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6e86c36d-22a6-4e70-8664-b7fce6d95869-signing-key\") pod \"service-ca-568869fdc5-dkg2s\" (UID: \"6e86c36d-22a6-4e70-8664-b7fce6d95869\") " pod="openshift-service-ca/service-ca-568869fdc5-dkg2s" Sep 14 02:52:06.408401 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:06.408380 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/6e86c36d-22a6-4e70-8664-b7fce6d95869-signing-cabundle\") pod \"service-ca-568869fdc5-dkg2s\" (UID: \"6e86c36d-22a6-4e70-8664-b7fce6d95869\") " pod="openshift-service-ca/service-ca-568869fdc5-dkg2s" Sep 14 02:52:06.410262 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:06.410245 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/6e86c36d-22a6-4e70-8664-b7fce6d95869-signing-key\") pod \"service-ca-568869fdc5-dkg2s\" (UID: \"6e86c36d-22a6-4e70-8664-b7fce6d95869\") " pod="openshift-service-ca/service-ca-568869fdc5-dkg2s" Sep 14 02:52:06.418860 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:06.418834 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gc9d\" (UniqueName: \"kubernetes.io/projected/6e86c36d-22a6-4e70-8664-b7fce6d95869-kube-api-access-7gc9d\") pod \"service-ca-568869fdc5-dkg2s\" (UID: \"6e86c36d-22a6-4e70-8664-b7fce6d95869\") " pod="openshift-service-ca/service-ca-568869fdc5-dkg2s" Sep 14 02:52:06.543043 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:06.542965 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-dkg2s" Sep 14 02:52:06.819656 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:06.819580 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-dkg2s"] Sep 14 02:52:06.822774 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:52:06.822743 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e86c36d_22a6_4e70_8664_b7fce6d95869.slice/crio-a1affa32237817e14fb643db97c2e33bd24b52bc8b7bd0eef5d5e019051fc712 WatchSource:0}: Error finding container a1affa32237817e14fb643db97c2e33bd24b52bc8b7bd0eef5d5e019051fc712: Status 404 returned error can't find the container with id a1affa32237817e14fb643db97c2e33bd24b52bc8b7bd0eef5d5e019051fc712 Sep 14 02:52:07.113920 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:07.113822 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls\") pod \"dns-default-znkd5\" (UID: \"47b16be3-755a-4a89-a2ef-830c72498f05\") " pod="openshift-dns/dns-default-znkd5" Sep 14 02:52:07.113920 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:07.113881 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls\") pod \"image-registry-76c79f4bcc-mg4xz\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:52:07.114269 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:52:07.114000 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 02:52:07.114269 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:52:07.113999 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 02:52:07.114269 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:52:07.114012 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76c79f4bcc-mg4xz: secret "image-registry-tls" not found Sep 14 02:52:07.114269 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:52:07.114147 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls podName:47b16be3-755a-4a89-a2ef-830c72498f05 nodeName:}" failed. No retries permitted until 2026-09-14 02:53:11.114115429 +0000 UTC m=+162.843774535 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls") pod "dns-default-znkd5" (UID: "47b16be3-755a-4a89-a2ef-830c72498f05") : secret "dns-default-metrics-tls" not found Sep 14 02:52:07.114269 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:52:07.114218 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls podName:ba5bd25d-67d4-45cf-a2fa-e5169274172b nodeName:}" failed. No retries permitted until 2026-09-14 02:53:11.114201321 +0000 UTC m=+162.843860436 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls") pod "image-registry-76c79f4bcc-mg4xz" (UID: "ba5bd25d-67d4-45cf-a2fa-e5169274172b") : secret "image-registry-tls" not found Sep 14 02:52:07.117105 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:07.117081 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-dkg2s" event={"ID":"6e86c36d-22a6-4e70-8664-b7fce6d95869","Type":"ContainerStarted","Data":"a1affa32237817e14fb643db97c2e33bd24b52bc8b7bd0eef5d5e019051fc712"} Sep 14 02:52:07.497172 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:07.497117 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:07.696038 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:07.695998 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:07.813273 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:07.813184 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:09.123126 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:09.123096 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-dkg2s" event={"ID":"6e86c36d-22a6-4e70-8664-b7fce6d95869","Type":"ContainerStarted","Data":"5467860ccbafcc7ff8310fa76c5571b1bd98d22d4e1dea932f45a2d15ce49830"} Sep 14 02:52:09.147530 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:09.147458 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-dkg2s" podStartSLOduration=1.016645277 podStartE2EDuration="3.147443238s" totalImagesPullingTime="2.130797961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:09.147065535 +0000 UTC m=+100.876724650" watchObservedRunningTime="2026-09-14 02:52:09.147443238 +0000 UTC m=+100.877102353" Sep 14 02:52:09.653661 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:09.653613 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:10.583728 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:10.583699 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xgj42_06330f4e-f78f-49f6-8c53-9565d0364232/dns-node-resolver/0.log" Sep 14 02:52:11.055526 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:11.055497 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5hcgk" Sep 14 02:52:11.384243 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:11.384164 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tjwgv_58a26876-794d-43a3-9b28-d8375839bbcf/node-ca/0.log" Sep 14 02:52:11.523984 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:11.523944 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:11.624138 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:11.624103 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:11.744188 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:11.744152 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:11.856564 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:11.856527 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:11.957125 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:11.957052 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:12.053456 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:12.053314 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:12.183776 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:12.183744 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8px75_1b6e2fea-3457-4b62-ba40-0254ed00fb1e/serve-healthcheck-canary/0.log" Sep 14 02:52:12.216440 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:12.216408 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:13.418413 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:13.418381 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:13.612573 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:13.612537 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:13.810433 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:13.810318 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:13.955835 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:13.955796 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:14.028664 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:14.028630 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:14.219647 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:14.219615 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:15.155238 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:15.155201 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:15.257437 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:15.257382 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:15.421353 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:15.421264 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:15.614160 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:15.614122 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:15.815461 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:15.815390 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:16.013967 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:16.013932 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:16.077969 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:16.077876 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:16.213528 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:16.213488 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:17.412707 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:17.412664 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:17.484646 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:17.484618 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:17.553288 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:17.553256 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:17.657274 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:17.657241 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:17.828725 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:17.828631 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:18.025726 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:18.025693 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:18.225920 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:18.225874 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:18.586443 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:18.586356 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:18.701299 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:18.701268 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:19.215548 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:19.215513 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:19.623967 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:19.623865 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:19.823842 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:19.823813 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:20.023710 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:20.023664 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:20.225044 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:20.225000 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:21.425385 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:21.425351 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:21.645661 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:21.645624 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:21.812081 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:21.811984 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:22.014989 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:22.014954 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:22.226150 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:22.226111 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:24.890891 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:24.890852 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:24.995242 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:24.995195 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:25.067962 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:25.067926 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:25.202274 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:25.202244 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:25.395310 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:25.395276 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:25.574612 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:25.574522 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:25.675554 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:25.675523 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:26.025513 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.025461 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:26.077396 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.077366 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-587665c8bb-9sq2v"] Sep 14 02:52:26.080261 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.080237 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kfljj"] Sep 14 02:52:26.080408 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.080388 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-587665c8bb-9sq2v" Sep 14 02:52:26.082816 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.082786 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 14 02:52:26.083030 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.083016 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-pfhbh\"" Sep 14 02:52:26.083150 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.083135 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kfljj" Sep 14 02:52:26.083581 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.083560 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 14 02:52:26.083777 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.083616 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 14 02:52:26.083777 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.083642 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 14 02:52:26.086264 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.086245 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pmfgr\"" Sep 14 02:52:26.086658 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.086633 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 14 02:52:26.086928 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.086915 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 14 02:52:26.087008 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.086952 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 14 02:52:26.087817 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.087540 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 14 02:52:26.098765 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.098736 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-587665c8bb-9sq2v"] Sep 14 02:52:26.107367 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.107346 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kfljj"] Sep 14 02:52:26.147260 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.147231 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:26.192521 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.192500 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-dqfwl"] Sep 14 02:52:26.195300 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.195278 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-dqfwl" Sep 14 02:52:26.195774 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.195754 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-dtzk7"] Sep 14 02:52:26.201855 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.201834 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-dtzk7" Sep 14 02:52:26.208133 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.208004 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 14 02:52:26.208717 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.208701 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-vk64q\"" Sep 14 02:52:26.209811 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.209795 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 14 02:52:26.211827 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.211811 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ftbnv\"" Sep 14 02:52:26.216129 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.216114 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 14 02:52:26.217212 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.217198 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 14 02:52:26.232392 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.232375 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-dqfwl"] Sep 14 02:52:26.253681 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.253654 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/83942481-0d5f-46ca-b957-7a5ba2066b7d-crio-socket\") pod \"insights-runtime-extractor-kfljj\" (UID: \"83942481-0d5f-46ca-b957-7a5ba2066b7d\") " pod="openshift-insights/insights-runtime-extractor-kfljj" Sep 14 02:52:26.253769 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.253683 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/83942481-0d5f-46ca-b957-7a5ba2066b7d-data-volume\") pod \"insights-runtime-extractor-kfljj\" (UID: \"83942481-0d5f-46ca-b957-7a5ba2066b7d\") " pod="openshift-insights/insights-runtime-extractor-kfljj" Sep 14 02:52:26.253769 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.253717 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83942481-0d5f-46ca-b957-7a5ba2066b7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kfljj\" (UID: \"83942481-0d5f-46ca-b957-7a5ba2066b7d\") " pod="openshift-insights/insights-runtime-extractor-kfljj" Sep 14 02:52:26.253769 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.253740 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83942481-0d5f-46ca-b957-7a5ba2066b7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfljj\" (UID: \"83942481-0d5f-46ca-b957-7a5ba2066b7d\") " pod="openshift-insights/insights-runtime-extractor-kfljj" Sep 14 02:52:26.253885 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.253785 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9vk9\" (UniqueName: \"kubernetes.io/projected/83942481-0d5f-46ca-b957-7a5ba2066b7d-kube-api-access-l9vk9\") pod \"insights-runtime-extractor-kfljj\" (UID: \"83942481-0d5f-46ca-b957-7a5ba2066b7d\") " pod="openshift-insights/insights-runtime-extractor-kfljj" Sep 14 02:52:26.253885 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.253826 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/270b35e0-c6cc-4577-a9b9-618126ea27db-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-587665c8bb-9sq2v\" (UID: \"270b35e0-c6cc-4577-a9b9-618126ea27db\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-587665c8bb-9sq2v" Sep 14 02:52:26.253885 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.253859 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6k5v\" (UniqueName: \"kubernetes.io/projected/270b35e0-c6cc-4577-a9b9-618126ea27db-kube-api-access-q6k5v\") pod \"managed-serviceaccount-addon-agent-587665c8bb-9sq2v\" (UID: \"270b35e0-c6cc-4577-a9b9-618126ea27db\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-587665c8bb-9sq2v" Sep 14 02:52:26.267588 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.267569 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-dtzk7"] Sep 14 02:52:26.298369 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.298316 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:26.354177 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.354152 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/83942481-0d5f-46ca-b957-7a5ba2066b7d-crio-socket\") pod \"insights-runtime-extractor-kfljj\" (UID: \"83942481-0d5f-46ca-b957-7a5ba2066b7d\") " pod="openshift-insights/insights-runtime-extractor-kfljj" Sep 14 02:52:26.354177 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.354177 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/83942481-0d5f-46ca-b957-7a5ba2066b7d-data-volume\") pod \"insights-runtime-extractor-kfljj\" (UID: \"83942481-0d5f-46ca-b957-7a5ba2066b7d\") " pod="openshift-insights/insights-runtime-extractor-kfljj" Sep 14 02:52:26.354321 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.354200 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nctnv\" (UniqueName: \"kubernetes.io/projected/06abfff7-30d4-4770-994a-ee2fd0215cd5-kube-api-access-nctnv\") pod \"downloads-76f99584f4-dqfwl\" (UID: \"06abfff7-30d4-4770-994a-ee2fd0215cd5\") " pod="openshift-console/downloads-76f99584f4-dqfwl" Sep 14 02:52:26.354321 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.354232 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83942481-0d5f-46ca-b957-7a5ba2066b7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kfljj\" (UID: \"83942481-0d5f-46ca-b957-7a5ba2066b7d\") " pod="openshift-insights/insights-runtime-extractor-kfljj" Sep 14 02:52:26.354321 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.354273 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83942481-0d5f-46ca-b957-7a5ba2066b7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfljj\" (UID: \"83942481-0d5f-46ca-b957-7a5ba2066b7d\") " pod="openshift-insights/insights-runtime-extractor-kfljj" Sep 14 02:52:26.354321 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.354307 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9vk9\" (UniqueName: \"kubernetes.io/projected/83942481-0d5f-46ca-b957-7a5ba2066b7d-kube-api-access-l9vk9\") pod \"insights-runtime-extractor-kfljj\" (UID: \"83942481-0d5f-46ca-b957-7a5ba2066b7d\") " pod="openshift-insights/insights-runtime-extractor-kfljj" Sep 14 02:52:26.354321 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.354273 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/83942481-0d5f-46ca-b957-7a5ba2066b7d-crio-socket\") pod \"insights-runtime-extractor-kfljj\" (UID: \"83942481-0d5f-46ca-b957-7a5ba2066b7d\") " pod="openshift-insights/insights-runtime-extractor-kfljj" Sep 14 02:52:26.354585 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.354330 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/270b35e0-c6cc-4577-a9b9-618126ea27db-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-587665c8bb-9sq2v\" (UID: \"270b35e0-c6cc-4577-a9b9-618126ea27db\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-587665c8bb-9sq2v" Sep 14 02:52:26.354585 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.354362 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6k5v\" (UniqueName: \"kubernetes.io/projected/270b35e0-c6cc-4577-a9b9-618126ea27db-kube-api-access-q6k5v\") pod \"managed-serviceaccount-addon-agent-587665c8bb-9sq2v\" (UID: \"270b35e0-c6cc-4577-a9b9-618126ea27db\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-587665c8bb-9sq2v" Sep 14 02:52:26.354585 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.354418 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trmmp\" (UniqueName: \"kubernetes.io/projected/e1f8f137-bf5f-4e2d-8833-00b4d255aa9c-kube-api-access-trmmp\") pod \"migrator-8b4d697c5-dtzk7\" (UID: \"e1f8f137-bf5f-4e2d-8833-00b4d255aa9c\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-dtzk7" Sep 14 02:52:26.354585 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.354580 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/83942481-0d5f-46ca-b957-7a5ba2066b7d-data-volume\") pod \"insights-runtime-extractor-kfljj\" (UID: \"83942481-0d5f-46ca-b957-7a5ba2066b7d\") " pod="openshift-insights/insights-runtime-extractor-kfljj" Sep 14 02:52:26.354857 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.354837 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83942481-0d5f-46ca-b957-7a5ba2066b7d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kfljj\" (UID: \"83942481-0d5f-46ca-b957-7a5ba2066b7d\") " pod="openshift-insights/insights-runtime-extractor-kfljj" Sep 14 02:52:26.356658 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.356639 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83942481-0d5f-46ca-b957-7a5ba2066b7d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kfljj\" (UID: \"83942481-0d5f-46ca-b957-7a5ba2066b7d\") " pod="openshift-insights/insights-runtime-extractor-kfljj" Sep 14 02:52:26.357085 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.357064 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/270b35e0-c6cc-4577-a9b9-618126ea27db-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-587665c8bb-9sq2v\" (UID: \"270b35e0-c6cc-4577-a9b9-618126ea27db\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-587665c8bb-9sq2v" Sep 14 02:52:26.364409 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.364388 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9vk9\" (UniqueName: \"kubernetes.io/projected/83942481-0d5f-46ca-b957-7a5ba2066b7d-kube-api-access-l9vk9\") pod \"insights-runtime-extractor-kfljj\" (UID: \"83942481-0d5f-46ca-b957-7a5ba2066b7d\") " pod="openshift-insights/insights-runtime-extractor-kfljj" Sep 14 02:52:26.364559 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.364394 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6k5v\" (UniqueName: \"kubernetes.io/projected/270b35e0-c6cc-4577-a9b9-618126ea27db-kube-api-access-q6k5v\") pod \"managed-serviceaccount-addon-agent-587665c8bb-9sq2v\" (UID: \"270b35e0-c6cc-4577-a9b9-618126ea27db\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-587665c8bb-9sq2v" Sep 14 02:52:26.389625 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.389600 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:26.399408 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.399378 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-587665c8bb-9sq2v" Sep 14 02:52:26.405210 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.405193 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kfljj" Sep 14 02:52:26.455184 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.455021 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trmmp\" (UniqueName: \"kubernetes.io/projected/e1f8f137-bf5f-4e2d-8833-00b4d255aa9c-kube-api-access-trmmp\") pod \"migrator-8b4d697c5-dtzk7\" (UID: \"e1f8f137-bf5f-4e2d-8833-00b4d255aa9c\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-dtzk7" Sep 14 02:52:26.455184 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.455109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nctnv\" (UniqueName: \"kubernetes.io/projected/06abfff7-30d4-4770-994a-ee2fd0215cd5-kube-api-access-nctnv\") pod \"downloads-76f99584f4-dqfwl\" (UID: \"06abfff7-30d4-4770-994a-ee2fd0215cd5\") " pod="openshift-console/downloads-76f99584f4-dqfwl" Sep 14 02:52:26.466999 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.466948 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nctnv\" (UniqueName: \"kubernetes.io/projected/06abfff7-30d4-4770-994a-ee2fd0215cd5-kube-api-access-nctnv\") pod \"downloads-76f99584f4-dqfwl\" (UID: \"06abfff7-30d4-4770-994a-ee2fd0215cd5\") " pod="openshift-console/downloads-76f99584f4-dqfwl" Sep 14 02:52:26.467128 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.467109 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trmmp\" (UniqueName: \"kubernetes.io/projected/e1f8f137-bf5f-4e2d-8833-00b4d255aa9c-kube-api-access-trmmp\") pod \"migrator-8b4d697c5-dtzk7\" (UID: \"e1f8f137-bf5f-4e2d-8833-00b4d255aa9c\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-dtzk7" Sep 14 02:52:26.504529 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.504502 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-dqfwl" Sep 14 02:52:26.510437 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.510418 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-dtzk7" Sep 14 02:52:26.580308 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.580230 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:26.702884 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:52:26.702855 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod270b35e0_c6cc_4577_a9b9_618126ea27db.slice/crio-6e207893bb6f311502aac8e9b71a314748218c0d20abe2b49190abcce001656d WatchSource:0}: Error finding container 6e207893bb6f311502aac8e9b71a314748218c0d20abe2b49190abcce001656d: Status 404 returned error can't find the container with id 6e207893bb6f311502aac8e9b71a314748218c0d20abe2b49190abcce001656d Sep 14 02:52:26.706227 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:52:26.706205 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83942481_0d5f_46ca_b957_7a5ba2066b7d.slice/crio-ab32b8e5c33e1cbd52c95eeee67643cb8cd4360ec35a8deb4c40b4b79740cb4d WatchSource:0}: Error finding container ab32b8e5c33e1cbd52c95eeee67643cb8cd4360ec35a8deb4c40b4b79740cb4d: Status 404 returned error can't find the container with id ab32b8e5c33e1cbd52c95eeee67643cb8cd4360ec35a8deb4c40b4b79740cb4d Sep 14 02:52:26.711123 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.711101 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-587665c8bb-9sq2v"] Sep 14 02:52:26.712305 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.712283 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kfljj"] Sep 14 02:52:26.782779 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.782752 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-dqfwl"] Sep 14 02:52:26.785846 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:52:26.785823 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06abfff7_30d4_4770_994a_ee2fd0215cd5.slice/crio-e004655bc179852c74b9a888a8c6bc17aef9d85b0fb3f21c6d9ad62a741ab00c WatchSource:0}: Error finding container e004655bc179852c74b9a888a8c6bc17aef9d85b0fb3f21c6d9ad62a741ab00c: Status 404 returned error can't find the container with id e004655bc179852c74b9a888a8c6bc17aef9d85b0fb3f21c6d9ad62a741ab00c Sep 14 02:52:26.789181 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.789141 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-dtzk7"] Sep 14 02:52:26.791849 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:52:26.791826 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f8f137_bf5f_4e2d_8833_00b4d255aa9c.slice/crio-1086d733a83267346fe4c4f571f820b8ee0716c128ad61a8815cdd047ec60166 WatchSource:0}: Error finding container 1086d733a83267346fe4c4f571f820b8ee0716c128ad61a8815cdd047ec60166: Status 404 returned error can't find the container with id 1086d733a83267346fe4c4f571f820b8ee0716c128ad61a8815cdd047ec60166 Sep 14 02:52:26.797137 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.797109 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:26.892465 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:26.892385 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:27.173512 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:27.173406 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-dqfwl" event={"ID":"06abfff7-30d4-4770-994a-ee2fd0215cd5","Type":"ContainerStarted","Data":"e004655bc179852c74b9a888a8c6bc17aef9d85b0fb3f21c6d9ad62a741ab00c"} Sep 14 02:52:27.174521 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:27.174495 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-587665c8bb-9sq2v" event={"ID":"270b35e0-c6cc-4577-a9b9-618126ea27db","Type":"ContainerStarted","Data":"6e207893bb6f311502aac8e9b71a314748218c0d20abe2b49190abcce001656d"} Sep 14 02:52:27.175814 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:27.175796 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kfljj" event={"ID":"83942481-0d5f-46ca-b957-7a5ba2066b7d","Type":"ContainerStarted","Data":"0538ab718dc004be7b8731a4fd11c54b36bdd294aa32f4a61054210bc6d7f7a1"} Sep 14 02:52:27.175886 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:27.175831 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kfljj" event={"ID":"83942481-0d5f-46ca-b957-7a5ba2066b7d","Type":"ContainerStarted","Data":"ab32b8e5c33e1cbd52c95eeee67643cb8cd4360ec35a8deb4c40b4b79740cb4d"} Sep 14 02:52:27.176871 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:27.176853 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-dtzk7" event={"ID":"e1f8f137-bf5f-4e2d-8833-00b4d255aa9c","Type":"ContainerStarted","Data":"1086d733a83267346fe4c4f571f820b8ee0716c128ad61a8815cdd047ec60166"} Sep 14 02:52:27.213975 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:27.213935 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:27.302080 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:27.302032 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:28.185400 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:28.185305 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kfljj" event={"ID":"83942481-0d5f-46ca-b957-7a5ba2066b7d","Type":"ContainerStarted","Data":"6d2dfa5ca9a34177cc8e5a8b2ca4aa6a985822bf120b98b63587d9dc6d2dfc8a"} Sep 14 02:52:28.187641 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:28.187611 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-dtzk7" event={"ID":"e1f8f137-bf5f-4e2d-8833-00b4d255aa9c","Type":"ContainerStarted","Data":"c645691aaed7abc13dba1120dbe4c721b8293f41afda75e48df606c0a30ec423"} Sep 14 02:52:29.193314 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:29.193265 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-dtzk7" event={"ID":"e1f8f137-bf5f-4e2d-8833-00b4d255aa9c","Type":"ContainerStarted","Data":"535e3464c87e27f386ebb6edcfc2a1f37e796c83fb486b3224ae9f3b39c2b279"} Sep 14 02:52:29.209097 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:29.209046 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-dtzk7" podStartSLOduration=1.933266246 podStartE2EDuration="3.209028385s" totalImagesPullingTime="1.275762139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:29.207952636 +0000 UTC m=+120.937611752" watchObservedRunningTime="2026-09-14 02:52:29.209028385 +0000 UTC m=+120.938687500" Sep 14 02:52:29.292366 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:29.292335 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:29.391594 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:29.391554 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:30.199180 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:30.199146 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-587665c8bb-9sq2v" event={"ID":"270b35e0-c6cc-4577-a9b9-618126ea27db","Type":"ContainerStarted","Data":"8e3737502c662144bc3c6030f61ba8f8d43e9fc43dc016c4d4041b1acaec9956"} Sep 14 02:52:30.201861 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:30.201835 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kfljj" event={"ID":"83942481-0d5f-46ca-b957-7a5ba2066b7d","Type":"ContainerStarted","Data":"53bd7400cd75e4823ebf7c94554bdba1e740a3d708dce6ce8d4e0120f8e5d8a7"} Sep 14 02:52:30.214089 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:30.214044 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-587665c8bb-9sq2v" podStartSLOduration=0.80246921 podStartE2EDuration="4.214025298s" totalImagesPullingTime="3.411556088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:30.213010644 +0000 UTC m=+121.942669761" watchObservedRunningTime="2026-09-14 02:52:30.214025298 +0000 UTC m=+121.943684413" Sep 14 02:52:30.232061 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:30.232012 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kfljj" podStartSLOduration=0.941435742 podStartE2EDuration="4.231995595s" totalImagesPullingTime="3.290559853s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:30.231273001 +0000 UTC m=+121.960932135" watchObservedRunningTime="2026-09-14 02:52:30.231995595 +0000 UTC m=+121.961654711" Sep 14 02:52:33.815604 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:33.815465 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:33.828793 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:33.828764 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c9d6f945d-ph585"] Sep 14 02:52:33.832020 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:33.831991 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:52:33.835009 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:33.834981 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 14 02:52:33.835009 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:33.835000 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vw64g\"" Sep 14 02:52:33.835165 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:33.835033 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 14 02:52:33.835165 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:33.835067 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 14 02:52:33.835297 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:33.835284 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 14 02:52:33.835384 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:33.835369 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 14 02:52:33.844675 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:33.844652 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c9d6f945d-ph585"] Sep 14 02:52:33.925212 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:33.925184 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c09dff-03ba-4fac-ae74-a41b1812bf38-service-ca\") pod \"console-7c9d6f945d-ph585\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:52:33.925372 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:33.925220 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5c09dff-03ba-4fac-ae74-a41b1812bf38-console-oauth-config\") pod \"console-7c9d6f945d-ph585\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:52:33.925372 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:33.925246 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c09dff-03ba-4fac-ae74-a41b1812bf38-console-serving-cert\") pod \"console-7c9d6f945d-ph585\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:52:33.925372 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:33.925357 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwtwm\" (UniqueName: \"kubernetes.io/projected/d5c09dff-03ba-4fac-ae74-a41b1812bf38-kube-api-access-xwtwm\") pod \"console-7c9d6f945d-ph585\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:52:33.925507 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:33.925386 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5c09dff-03ba-4fac-ae74-a41b1812bf38-oauth-serving-cert\") pod \"console-7c9d6f945d-ph585\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:52:33.925507 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:33.925429 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5c09dff-03ba-4fac-ae74-a41b1812bf38-console-config\") pod \"console-7c9d6f945d-ph585\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:52:34.026191 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:34.026156 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c09dff-03ba-4fac-ae74-a41b1812bf38-service-ca\") pod \"console-7c9d6f945d-ph585\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:52:34.026191 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:34.026194 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5c09dff-03ba-4fac-ae74-a41b1812bf38-console-oauth-config\") pod \"console-7c9d6f945d-ph585\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:52:34.026424 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:34.026213 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c09dff-03ba-4fac-ae74-a41b1812bf38-console-serving-cert\") pod \"console-7c9d6f945d-ph585\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:52:34.026424 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:34.026275 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwtwm\" (UniqueName: \"kubernetes.io/projected/d5c09dff-03ba-4fac-ae74-a41b1812bf38-kube-api-access-xwtwm\") pod \"console-7c9d6f945d-ph585\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:52:34.026424 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:34.026313 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5c09dff-03ba-4fac-ae74-a41b1812bf38-oauth-serving-cert\") pod \"console-7c9d6f945d-ph585\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:52:34.026424 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:34.026364 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5c09dff-03ba-4fac-ae74-a41b1812bf38-console-config\") pod \"console-7c9d6f945d-ph585\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:52:34.027027 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:34.027001 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5c09dff-03ba-4fac-ae74-a41b1812bf38-console-config\") pod \"console-7c9d6f945d-ph585\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:52:34.027150 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:34.027001 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c09dff-03ba-4fac-ae74-a41b1812bf38-service-ca\") pod \"console-7c9d6f945d-ph585\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:52:34.027150 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:34.027044 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5c09dff-03ba-4fac-ae74-a41b1812bf38-oauth-serving-cert\") pod \"console-7c9d6f945d-ph585\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:52:34.029433 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:34.029412 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5c09dff-03ba-4fac-ae74-a41b1812bf38-console-oauth-config\") pod \"console-7c9d6f945d-ph585\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:52:34.029541 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:34.029513 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c09dff-03ba-4fac-ae74-a41b1812bf38-console-serving-cert\") pod \"console-7c9d6f945d-ph585\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:52:34.034927 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:34.034900 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwtwm\" (UniqueName: \"kubernetes.io/projected/d5c09dff-03ba-4fac-ae74-a41b1812bf38-kube-api-access-xwtwm\") pod \"console-7c9d6f945d-ph585\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:52:34.142360 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:34.142278 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:52:34.179144 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:34.179110 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:34.276598 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:34.276559 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:34.386306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:34.386260 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:34.426671 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:34.426650 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c9d6f945d-ph585"] Sep 14 02:52:34.429288 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:52:34.429255 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c09dff_03ba_4fac_ae74_a41b1812bf38.slice/crio-f6f5d060087f3a9089401a09720804e0436c8485b0559ca1bb63dfc80309b9c9 WatchSource:0}: Error finding container f6f5d060087f3a9089401a09720804e0436c8485b0559ca1bb63dfc80309b9c9: Status 404 returned error can't find the container with id f6f5d060087f3a9089401a09720804e0436c8485b0559ca1bb63dfc80309b9c9 Sep 14 02:52:34.482589 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:34.482530 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:34.913286 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:34.913250 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:35.221982 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:35.221900 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9d6f945d-ph585" event={"ID":"d5c09dff-03ba-4fac-ae74-a41b1812bf38","Type":"ContainerStarted","Data":"f6f5d060087f3a9089401a09720804e0436c8485b0559ca1bb63dfc80309b9c9"} Sep 14 02:52:35.840041 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:35.840010 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:35.940141 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:35.940103 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:38.402035 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:38.401998 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:38.485931 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:38.485895 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:38.668009 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:38.667925 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs\") pod \"network-metrics-daemon-jrk97\" (UID: \"441362ed-9e44-4875-9e60-ff99bf8fa8b1\") " pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:52:38.670725 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:38.670701 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/441362ed-9e44-4875-9e60-ff99bf8fa8b1-metrics-certs\") pod \"network-metrics-daemon-jrk97\" (UID: \"441362ed-9e44-4875-9e60-ff99bf8fa8b1\") " pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:52:38.867058 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:38.867027 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mk9cr\"" Sep 14 02:52:38.874956 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:38.874924 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jrk97" Sep 14 02:52:38.947687 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:38.947511 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:41.170345 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:41.170295 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:42.214128 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:42.214063 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:42.274492 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:42.274442 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:42.611799 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:42.611707 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:42.756204 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:42.756174 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:42.896093 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:42.896058 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:42.936041 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:42.935954 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-v8xq9"] Sep 14 02:52:42.940424 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:42.939809 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:42.944383 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:42.943727 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 14 02:52:42.944383 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:42.943876 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 14 02:52:42.944383 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:42.943934 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 14 02:52:42.944383 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:42.944062 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 14 02:52:42.944383 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:42.944237 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 14 02:52:42.944383 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:42.944290 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 14 02:52:42.944757 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:42.944698 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qnmp8\"" Sep 14 02:52:42.966176 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:42.966128 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd"] Sep 14 02:52:42.970324 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:42.970304 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" Sep 14 02:52:42.976309 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:42.976288 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 14 02:52:42.976646 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:42.976629 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-p45zg\"" Sep 14 02:52:42.976915 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:42.976897 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 14 02:52:42.977661 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:42.977642 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 14 02:52:43.005985 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.004500 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42ab45b2-5436-4392-b8a1-000f53af76df-sys\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.005985 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.004533 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42ab45b2-5436-4392-b8a1-000f53af76df-root\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.005985 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.004566 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42ab45b2-5436-4392-b8a1-000f53af76df-node-exporter-tls\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.005985 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.004600 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42ab45b2-5436-4392-b8a1-000f53af76df-node-exporter-textfile\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.005985 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.004623 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8lrk\" (UniqueName: \"kubernetes.io/projected/42ab45b2-5436-4392-b8a1-000f53af76df-kube-api-access-n8lrk\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.005985 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.004666 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42ab45b2-5436-4392-b8a1-000f53af76df-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.005985 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.004693 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42ab45b2-5436-4392-b8a1-000f53af76df-node-exporter-accelerators-collector-config\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.005985 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.004722 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42ab45b2-5436-4392-b8a1-000f53af76df-node-exporter-wtmp\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.005985 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.004747 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42ab45b2-5436-4392-b8a1-000f53af76df-metrics-client-ca\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.005985 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.005923 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd"] Sep 14 02:52:43.050738 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.050704 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:43.106508 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.105846 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42ab45b2-5436-4392-b8a1-000f53af76df-metrics-client-ca\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.106508 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.105909 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/049d80ec-0298-4f01-a744-5bfcd8ea6bcf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-w24sd\" (UID: \"049d80ec-0298-4f01-a744-5bfcd8ea6bcf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" Sep 14 02:52:43.106508 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.105956 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42ab45b2-5436-4392-b8a1-000f53af76df-sys\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.106508 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.105977 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42ab45b2-5436-4392-b8a1-000f53af76df-root\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.106508 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.106005 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42ab45b2-5436-4392-b8a1-000f53af76df-node-exporter-tls\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.106508 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.106034 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42ab45b2-5436-4392-b8a1-000f53af76df-node-exporter-textfile\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.106508 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.106103 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42ab45b2-5436-4392-b8a1-000f53af76df-sys\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.107032 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.106988 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42ab45b2-5436-4392-b8a1-000f53af76df-root\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.107111 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:52:43.107070 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 14 02:52:43.107163 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:52:43.107135 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42ab45b2-5436-4392-b8a1-000f53af76df-node-exporter-tls podName:42ab45b2-5436-4392-b8a1-000f53af76df nodeName:}" failed. No retries permitted until 2026-09-14 02:52:43.607121788 +0000 UTC m=+135.336780882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/42ab45b2-5436-4392-b8a1-000f53af76df-node-exporter-tls") pod "node-exporter-v8xq9" (UID: "42ab45b2-5436-4392-b8a1-000f53af76df") : secret "node-exporter-tls" not found Sep 14 02:52:43.108624 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.107407 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42ab45b2-5436-4392-b8a1-000f53af76df-node-exporter-textfile\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.108624 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.107692 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42ab45b2-5436-4392-b8a1-000f53af76df-metrics-client-ca\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.108624 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.107754 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8lrk\" (UniqueName: \"kubernetes.io/projected/42ab45b2-5436-4392-b8a1-000f53af76df-kube-api-access-n8lrk\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.108624 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.107784 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/049d80ec-0298-4f01-a744-5bfcd8ea6bcf-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-w24sd\" (UID: \"049d80ec-0298-4f01-a744-5bfcd8ea6bcf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" Sep 14 02:52:43.108624 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.107813 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/049d80ec-0298-4f01-a744-5bfcd8ea6bcf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-w24sd\" (UID: \"049d80ec-0298-4f01-a744-5bfcd8ea6bcf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" Sep 14 02:52:43.108624 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.107837 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkq62\" (UniqueName: \"kubernetes.io/projected/049d80ec-0298-4f01-a744-5bfcd8ea6bcf-kube-api-access-mkq62\") pod \"kube-state-metrics-59fbdd77d-w24sd\" (UID: \"049d80ec-0298-4f01-a744-5bfcd8ea6bcf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" Sep 14 02:52:43.108624 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.107882 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42ab45b2-5436-4392-b8a1-000f53af76df-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.108624 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.107913 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42ab45b2-5436-4392-b8a1-000f53af76df-node-exporter-accelerators-collector-config\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.108624 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.107936 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/049d80ec-0298-4f01-a744-5bfcd8ea6bcf-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-w24sd\" (UID: \"049d80ec-0298-4f01-a744-5bfcd8ea6bcf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" Sep 14 02:52:43.108624 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.107962 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/049d80ec-0298-4f01-a744-5bfcd8ea6bcf-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-w24sd\" (UID: \"049d80ec-0298-4f01-a744-5bfcd8ea6bcf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" Sep 14 02:52:43.108624 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.107990 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42ab45b2-5436-4392-b8a1-000f53af76df-node-exporter-wtmp\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.108624 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.108130 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42ab45b2-5436-4392-b8a1-000f53af76df-node-exporter-wtmp\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.109179 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.109163 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42ab45b2-5436-4392-b8a1-000f53af76df-node-exporter-accelerators-collector-config\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.112008 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.111986 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42ab45b2-5436-4392-b8a1-000f53af76df-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.120057 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.120035 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8lrk\" (UniqueName: \"kubernetes.io/projected/42ab45b2-5436-4392-b8a1-000f53af76df-kube-api-access-n8lrk\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.138222 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.138141 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:43.210983 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.208954 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/049d80ec-0298-4f01-a744-5bfcd8ea6bcf-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-w24sd\" (UID: \"049d80ec-0298-4f01-a744-5bfcd8ea6bcf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" Sep 14 02:52:43.210983 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.208998 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/049d80ec-0298-4f01-a744-5bfcd8ea6bcf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-w24sd\" (UID: \"049d80ec-0298-4f01-a744-5bfcd8ea6bcf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" Sep 14 02:52:43.210983 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.209024 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkq62\" (UniqueName: \"kubernetes.io/projected/049d80ec-0298-4f01-a744-5bfcd8ea6bcf-kube-api-access-mkq62\") pod \"kube-state-metrics-59fbdd77d-w24sd\" (UID: \"049d80ec-0298-4f01-a744-5bfcd8ea6bcf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" Sep 14 02:52:43.210983 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.209068 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/049d80ec-0298-4f01-a744-5bfcd8ea6bcf-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-w24sd\" (UID: \"049d80ec-0298-4f01-a744-5bfcd8ea6bcf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" Sep 14 02:52:43.210983 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.209095 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/049d80ec-0298-4f01-a744-5bfcd8ea6bcf-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-w24sd\" (UID: \"049d80ec-0298-4f01-a744-5bfcd8ea6bcf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" Sep 14 02:52:43.210983 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.209153 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/049d80ec-0298-4f01-a744-5bfcd8ea6bcf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-w24sd\" (UID: \"049d80ec-0298-4f01-a744-5bfcd8ea6bcf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" Sep 14 02:52:43.210983 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.209987 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/049d80ec-0298-4f01-a744-5bfcd8ea6bcf-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-w24sd\" (UID: \"049d80ec-0298-4f01-a744-5bfcd8ea6bcf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" Sep 14 02:52:43.210983 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:52:43.210085 2564 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 14 02:52:43.210983 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:52:43.210146 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/049d80ec-0298-4f01-a744-5bfcd8ea6bcf-kube-state-metrics-tls podName:049d80ec-0298-4f01-a744-5bfcd8ea6bcf nodeName:}" failed. No retries permitted until 2026-09-14 02:52:43.710128887 +0000 UTC m=+135.439787995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/049d80ec-0298-4f01-a744-5bfcd8ea6bcf-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-w24sd" (UID: "049d80ec-0298-4f01-a744-5bfcd8ea6bcf") : secret "kube-state-metrics-tls" not found Sep 14 02:52:43.210983 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.210423 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/049d80ec-0298-4f01-a744-5bfcd8ea6bcf-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-w24sd\" (UID: \"049d80ec-0298-4f01-a744-5bfcd8ea6bcf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" Sep 14 02:52:43.210983 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.210878 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/049d80ec-0298-4f01-a744-5bfcd8ea6bcf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-w24sd\" (UID: \"049d80ec-0298-4f01-a744-5bfcd8ea6bcf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" Sep 14 02:52:43.218966 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.218904 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/049d80ec-0298-4f01-a744-5bfcd8ea6bcf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-w24sd\" (UID: \"049d80ec-0298-4f01-a744-5bfcd8ea6bcf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" Sep 14 02:52:43.219332 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.219113 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkq62\" (UniqueName: \"kubernetes.io/projected/049d80ec-0298-4f01-a744-5bfcd8ea6bcf-kube-api-access-mkq62\") pod \"kube-state-metrics-59fbdd77d-w24sd\" (UID: \"049d80ec-0298-4f01-a744-5bfcd8ea6bcf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" Sep 14 02:52:43.236961 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.236889 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:43.341498 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.341438 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:43.460401 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.460301 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:43.556399 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.556302 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:43.613092 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.613052 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42ab45b2-5436-4392-b8a1-000f53af76df-node-exporter-tls\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.615788 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.615764 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42ab45b2-5436-4392-b8a1-000f53af76df-node-exporter-tls\") pod \"node-exporter-v8xq9\" (UID: \"42ab45b2-5436-4392-b8a1-000f53af76df\") " pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.714374 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.714339 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/049d80ec-0298-4f01-a744-5bfcd8ea6bcf-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-w24sd\" (UID: \"049d80ec-0298-4f01-a744-5bfcd8ea6bcf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" Sep 14 02:52:43.716858 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.716823 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/049d80ec-0298-4f01-a744-5bfcd8ea6bcf-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-w24sd\" (UID: \"049d80ec-0298-4f01-a744-5bfcd8ea6bcf\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" Sep 14 02:52:43.720119 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.720093 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:43.848787 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.848691 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:43.852802 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.852765 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-v8xq9" Sep 14 02:52:43.882039 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.882013 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" Sep 14 02:52:43.946038 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:43.945993 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:44.039872 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.039841 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 02:52:44.044189 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.044165 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.053837 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.053769 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 14 02:52:44.053837 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.053826 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 14 02:52:44.054011 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.053857 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 14 02:52:44.054763 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.054598 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 14 02:52:44.054763 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.054601 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 14 02:52:44.054763 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.054652 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-spl6p\"" Sep 14 02:52:44.054763 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.054606 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 14 02:52:44.057795 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.057769 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 14 02:52:44.057888 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.057811 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 14 02:52:44.057888 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.057827 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 14 02:52:44.073340 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.073317 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 02:52:44.076784 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.076760 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:44.117794 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.117722 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c5693c3-3949-4015-9a7b-f5451574e59c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.117794 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.117776 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2c5693c3-3949-4015-9a7b-f5451574e59c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.117961 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.117815 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c5693c3-3949-4015-9a7b-f5451574e59c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.117961 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.117857 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c5693c3-3949-4015-9a7b-f5451574e59c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.117961 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.117899 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c5693c3-3949-4015-9a7b-f5451574e59c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.117961 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.117921 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c5693c3-3949-4015-9a7b-f5451574e59c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.118146 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.117964 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c5693c3-3949-4015-9a7b-f5451574e59c-config-out\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.118146 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.118006 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c5693c3-3949-4015-9a7b-f5451574e59c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.118146 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.118038 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnmg8\" (UniqueName: \"kubernetes.io/projected/2c5693c3-3949-4015-9a7b-f5451574e59c-kube-api-access-qnmg8\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.118146 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.118064 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2c5693c3-3949-4015-9a7b-f5451574e59c-config-volume\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.118146 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.118087 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2c5693c3-3949-4015-9a7b-f5451574e59c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.118371 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.118216 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c5693c3-3949-4015-9a7b-f5451574e59c-web-config\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.118371 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.118257 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2c5693c3-3949-4015-9a7b-f5451574e59c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.166084 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.166058 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:44.219330 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.219290 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c5693c3-3949-4015-9a7b-f5451574e59c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.219805 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.219342 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c5693c3-3949-4015-9a7b-f5451574e59c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.219805 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.219371 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c5693c3-3949-4015-9a7b-f5451574e59c-config-out\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.219805 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.219387 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c5693c3-3949-4015-9a7b-f5451574e59c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.219805 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.219415 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnmg8\" (UniqueName: \"kubernetes.io/projected/2c5693c3-3949-4015-9a7b-f5451574e59c-kube-api-access-qnmg8\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.219805 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.219441 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2c5693c3-3949-4015-9a7b-f5451574e59c-config-volume\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.219805 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.219462 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2c5693c3-3949-4015-9a7b-f5451574e59c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.219805 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.219531 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c5693c3-3949-4015-9a7b-f5451574e59c-web-config\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.219805 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.219563 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2c5693c3-3949-4015-9a7b-f5451574e59c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.219805 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.219611 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c5693c3-3949-4015-9a7b-f5451574e59c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.219805 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.219643 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2c5693c3-3949-4015-9a7b-f5451574e59c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.219805 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.219680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c5693c3-3949-4015-9a7b-f5451574e59c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.219805 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.219718 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c5693c3-3949-4015-9a7b-f5451574e59c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.220632 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.220600 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c5693c3-3949-4015-9a7b-f5451574e59c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.221019 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.220994 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2c5693c3-3949-4015-9a7b-f5451574e59c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.221409 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.221384 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c5693c3-3949-4015-9a7b-f5451574e59c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.222930 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.222887 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2c5693c3-3949-4015-9a7b-f5451574e59c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.223156 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.223129 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2c5693c3-3949-4015-9a7b-f5451574e59c-config-volume\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.223824 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.223799 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2c5693c3-3949-4015-9a7b-f5451574e59c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.224538 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.224519 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2c5693c3-3949-4015-9a7b-f5451574e59c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.225213 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.224891 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2c5693c3-3949-4015-9a7b-f5451574e59c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.225213 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.224963 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2c5693c3-3949-4015-9a7b-f5451574e59c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.225213 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.224993 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2c5693c3-3949-4015-9a7b-f5451574e59c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.225213 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.225166 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2c5693c3-3949-4015-9a7b-f5451574e59c-config-out\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.225213 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.225175 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2c5693c3-3949-4015-9a7b-f5451574e59c-web-config\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.227868 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.227849 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnmg8\" (UniqueName: \"kubernetes.io/projected/2c5693c3-3949-4015-9a7b-f5451574e59c-kube-api-access-qnmg8\") pod \"alertmanager-main-0\" (UID: \"2c5693c3-3949-4015-9a7b-f5451574e59c\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.354282 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.354249 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 02:52:44.385308 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:52:44.385269 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ab45b2_5436_4392_b8a1_000f53af76df.slice/crio-c8a26c13e654871dbce73c07777a045a10c1f32f5519f7ce51e22e879907b83a WatchSource:0}: Error finding container c8a26c13e654871dbce73c07777a045a10c1f32f5519f7ce51e22e879907b83a: Status 404 returned error can't find the container with id c8a26c13e654871dbce73c07777a045a10c1f32f5519f7ce51e22e879907b83a Sep 14 02:52:44.654592 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.654274 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jrk97"] Sep 14 02:52:44.658292 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:52:44.658257 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441362ed_9e44_4875_9e60_ff99bf8fa8b1.slice/crio-cc7d8577abf867242e58f40dca210c82812cdb8a4714b5cc3be5177ce5f68129 WatchSource:0}: Error finding container cc7d8577abf867242e58f40dca210c82812cdb8a4714b5cc3be5177ce5f68129: Status 404 returned error can't find the container with id cc7d8577abf867242e58f40dca210c82812cdb8a4714b5cc3be5177ce5f68129 Sep 14 02:52:44.679873 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.679845 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd"] Sep 14 02:52:44.682157 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:52:44.682127 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049d80ec_0298_4f01_a744_5bfcd8ea6bcf.slice/crio-6a9ab9386dd047c3cfb002559a4a2045066ff7ac731913a9a518120f70926ee3 WatchSource:0}: Error finding container 6a9ab9386dd047c3cfb002559a4a2045066ff7ac731913a9a518120f70926ee3: Status 404 returned error can't find the container with id 6a9ab9386dd047c3cfb002559a4a2045066ff7ac731913a9a518120f70926ee3 Sep 14 02:52:44.684134 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:44.684108 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 02:52:44.687315 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:52:44.687289 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5693c3_3949_4015_9a7b_f5451574e59c.slice/crio-4585e2c44381e581d727d7ea13d4bc15a18bf0bed58e5c7cca5ad15c6bd2550a WatchSource:0}: Error finding container 4585e2c44381e581d727d7ea13d4bc15a18bf0bed58e5c7cca5ad15c6bd2550a: Status 404 returned error can't find the container with id 4585e2c44381e581d727d7ea13d4bc15a18bf0bed58e5c7cca5ad15c6bd2550a Sep 14 02:52:45.224585 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:45.224392 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:45.264570 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:45.264320 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-dqfwl" event={"ID":"06abfff7-30d4-4770-994a-ee2fd0215cd5","Type":"ContainerStarted","Data":"cbfb10e23d5795933834308eece41e9056e14779cffdcf5088580e72700591f3"} Sep 14 02:52:45.266043 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:45.265806 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-dqfwl" Sep 14 02:52:45.267752 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:45.267707 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jrk97" event={"ID":"441362ed-9e44-4875-9e60-ff99bf8fa8b1","Type":"ContainerStarted","Data":"cc7d8577abf867242e58f40dca210c82812cdb8a4714b5cc3be5177ce5f68129"} Sep 14 02:52:45.269709 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:45.269686 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9d6f945d-ph585" event={"ID":"d5c09dff-03ba-4fac-ae74-a41b1812bf38","Type":"ContainerStarted","Data":"cfdcaa38aec9653c1c83b0b7c04a1347df1303ebac5643c4c9fb8daded4d0a8f"} Sep 14 02:52:45.271818 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:45.271773 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c5693c3-3949-4015-9a7b-f5451574e59c","Type":"ContainerStarted","Data":"4585e2c44381e581d727d7ea13d4bc15a18bf0bed58e5c7cca5ad15c6bd2550a"} Sep 14 02:52:45.273345 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:45.273281 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" event={"ID":"049d80ec-0298-4f01-a744-5bfcd8ea6bcf","Type":"ContainerStarted","Data":"6a9ab9386dd047c3cfb002559a4a2045066ff7ac731913a9a518120f70926ee3"} Sep 14 02:52:45.274974 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:45.274468 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v8xq9" event={"ID":"42ab45b2-5436-4392-b8a1-000f53af76df","Type":"ContainerStarted","Data":"c8a26c13e654871dbce73c07777a045a10c1f32f5519f7ce51e22e879907b83a"} Sep 14 02:52:45.284610 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:45.284221 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-dqfwl" podStartSLOduration=1.6357726000000001 podStartE2EDuration="19.284204132s" totalImagesPullingTime="17.648431532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:45.282824036 +0000 UTC m=+137.012483155" watchObservedRunningTime="2026-09-14 02:52:45.284204132 +0000 UTC m=+137.013863250" Sep 14 02:52:45.288217 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:45.288171 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-dqfwl" Sep 14 02:52:45.293757 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:45.293516 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:45.324921 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:45.323583 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c9d6f945d-ph585" podStartSLOduration=2.372121 podStartE2EDuration="12.323562787s" totalImagesPullingTime="9.951441787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:45.302892089 +0000 UTC m=+137.032551207" watchObservedRunningTime="2026-09-14 02:52:45.323562787 +0000 UTC m=+137.053221918" Sep 14 02:52:45.817961 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:45.817871 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:45.904253 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:45.904212 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:46.254300 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:46.254158 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c9d6f945d-ph585"] Sep 14 02:52:46.267159 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:46.267019 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:46.289510 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:46.288962 2564 generic.go:356] "Generic (PLEG): container finished" podID="42ab45b2-5436-4392-b8a1-000f53af76df" containerID="a98098f692ba405d19ab540254ab1cd4047ae4d69950ee1a28261125438e08ba" exitCode=0 Sep 14 02:52:46.289510 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:46.289195 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v8xq9" event={"ID":"42ab45b2-5436-4392-b8a1-000f53af76df","Type":"ContainerDied","Data":"a98098f692ba405d19ab540254ab1cd4047ae4d69950ee1a28261125438e08ba"} Sep 14 02:52:46.905118 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:46.905075 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:47.272534 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.272435 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-688bd89fc-t949d"] Sep 14 02:52:47.298304 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.298234 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-688bd89fc-t949d"] Sep 14 02:52:47.298450 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.298375 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.301882 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.301236 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 14 02:52:47.301882 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.301251 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 14 02:52:47.301882 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.301258 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-13gv2o43sk22b\"" Sep 14 02:52:47.301882 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.301409 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 14 02:52:47.301882 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.301538 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-v2dm4\"" Sep 14 02:52:47.301882 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.301750 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 14 02:52:47.355745 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.355704 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c02c878c-a03b-42ed-b288-6b044226cee0-metrics-server-audit-profiles\") pod \"metrics-server-688bd89fc-t949d\" (UID: \"c02c878c-a03b-42ed-b288-6b044226cee0\") " pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.355904 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.355772 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c02c878c-a03b-42ed-b288-6b044226cee0-audit-log\") pod \"metrics-server-688bd89fc-t949d\" (UID: \"c02c878c-a03b-42ed-b288-6b044226cee0\") " pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.355904 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.355876 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c02c878c-a03b-42ed-b288-6b044226cee0-secret-metrics-server-client-certs\") pod \"metrics-server-688bd89fc-t949d\" (UID: \"c02c878c-a03b-42ed-b288-6b044226cee0\") " pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.356015 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.355921 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c02c878c-a03b-42ed-b288-6b044226cee0-client-ca-bundle\") pod \"metrics-server-688bd89fc-t949d\" (UID: \"c02c878c-a03b-42ed-b288-6b044226cee0\") " pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.356015 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.355957 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkt5q\" (UniqueName: \"kubernetes.io/projected/c02c878c-a03b-42ed-b288-6b044226cee0-kube-api-access-bkt5q\") pod \"metrics-server-688bd89fc-t949d\" (UID: \"c02c878c-a03b-42ed-b288-6b044226cee0\") " pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.356110 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.356062 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c02c878c-a03b-42ed-b288-6b044226cee0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-688bd89fc-t949d\" (UID: \"c02c878c-a03b-42ed-b288-6b044226cee0\") " pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.356149 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.356114 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c02c878c-a03b-42ed-b288-6b044226cee0-secret-metrics-server-tls\") pod \"metrics-server-688bd89fc-t949d\" (UID: \"c02c878c-a03b-42ed-b288-6b044226cee0\") " pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.457306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.457268 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c02c878c-a03b-42ed-b288-6b044226cee0-secret-metrics-server-tls\") pod \"metrics-server-688bd89fc-t949d\" (UID: \"c02c878c-a03b-42ed-b288-6b044226cee0\") " pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.457488 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.457321 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c02c878c-a03b-42ed-b288-6b044226cee0-metrics-server-audit-profiles\") pod \"metrics-server-688bd89fc-t949d\" (UID: \"c02c878c-a03b-42ed-b288-6b044226cee0\") " pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.457488 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.457362 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c02c878c-a03b-42ed-b288-6b044226cee0-audit-log\") pod \"metrics-server-688bd89fc-t949d\" (UID: \"c02c878c-a03b-42ed-b288-6b044226cee0\") " pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.457488 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.457389 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c02c878c-a03b-42ed-b288-6b044226cee0-secret-metrics-server-client-certs\") pod \"metrics-server-688bd89fc-t949d\" (UID: \"c02c878c-a03b-42ed-b288-6b044226cee0\") " pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.457488 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.457413 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c02c878c-a03b-42ed-b288-6b044226cee0-client-ca-bundle\") pod \"metrics-server-688bd89fc-t949d\" (UID: \"c02c878c-a03b-42ed-b288-6b044226cee0\") " pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.457488 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.457460 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkt5q\" (UniqueName: \"kubernetes.io/projected/c02c878c-a03b-42ed-b288-6b044226cee0-kube-api-access-bkt5q\") pod \"metrics-server-688bd89fc-t949d\" (UID: \"c02c878c-a03b-42ed-b288-6b044226cee0\") " pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.457751 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.457561 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c02c878c-a03b-42ed-b288-6b044226cee0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-688bd89fc-t949d\" (UID: \"c02c878c-a03b-42ed-b288-6b044226cee0\") " pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.457902 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.457846 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c02c878c-a03b-42ed-b288-6b044226cee0-audit-log\") pod \"metrics-server-688bd89fc-t949d\" (UID: \"c02c878c-a03b-42ed-b288-6b044226cee0\") " pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.458265 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.458240 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c02c878c-a03b-42ed-b288-6b044226cee0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-688bd89fc-t949d\" (UID: \"c02c878c-a03b-42ed-b288-6b044226cee0\") " pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.458655 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.458601 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c02c878c-a03b-42ed-b288-6b044226cee0-metrics-server-audit-profiles\") pod \"metrics-server-688bd89fc-t949d\" (UID: \"c02c878c-a03b-42ed-b288-6b044226cee0\") " pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.460166 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.460146 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c02c878c-a03b-42ed-b288-6b044226cee0-secret-metrics-server-tls\") pod \"metrics-server-688bd89fc-t949d\" (UID: \"c02c878c-a03b-42ed-b288-6b044226cee0\") " pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.460269 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.460216 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c02c878c-a03b-42ed-b288-6b044226cee0-client-ca-bundle\") pod \"metrics-server-688bd89fc-t949d\" (UID: \"c02c878c-a03b-42ed-b288-6b044226cee0\") " pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.460803 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.460779 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c02c878c-a03b-42ed-b288-6b044226cee0-secret-metrics-server-client-certs\") pod \"metrics-server-688bd89fc-t949d\" (UID: \"c02c878c-a03b-42ed-b288-6b044226cee0\") " pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.466691 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.466647 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkt5q\" (UniqueName: \"kubernetes.io/projected/c02c878c-a03b-42ed-b288-6b044226cee0-kube-api-access-bkt5q\") pod \"metrics-server-688bd89fc-t949d\" (UID: \"c02c878c-a03b-42ed-b288-6b044226cee0\") " pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.587120 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.586930 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:47.610286 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.610262 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:52:47.687563 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:47.687526 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:48.135148 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.135114 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-688bd89fc-t949d"] Sep 14 02:52:48.151911 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.151752 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76c79f4bcc-mg4xz"] Sep 14 02:52:48.152159 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:52:48.152058 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" podUID="ba5bd25d-67d4-45cf-a2fa-e5169274172b" Sep 14 02:52:48.248533 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.248112 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-56b4cc6586-zhfrt"] Sep 14 02:52:48.265550 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.264468 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.269392 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.266698 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56b4cc6586-zhfrt"] Sep 14 02:52:48.269392 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.267620 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 14 02:52:48.270507 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.270181 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 14 02:52:48.270507 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.270390 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 14 02:52:48.274786 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.271123 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 14 02:52:48.274786 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.271364 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rtflm\"" Sep 14 02:52:48.274786 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.272950 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 14 02:52:48.285532 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.284776 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 14 02:52:48.305285 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.305257 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jrk97" event={"ID":"441362ed-9e44-4875-9e60-ff99bf8fa8b1","Type":"ContainerStarted","Data":"d987846c13ef6fbc36da5207b95f3a17a3295f676bd1b9b732c620484d3124e0"} Sep 14 02:52:48.305397 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.305337 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jrk97" event={"ID":"441362ed-9e44-4875-9e60-ff99bf8fa8b1","Type":"ContainerStarted","Data":"1261dc75710f2e84e1d0a5336470d9a54bd45f2a1d99fec4aed9384d7e2aa124"} Sep 14 02:52:48.308243 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.307419 2564 generic.go:356] "Generic (PLEG): container finished" podID="2c5693c3-3949-4015-9a7b-f5451574e59c" containerID="1f9cf377f97d7d63a451723694fc1b1cf331aa894737c3848b7114a399e62c3b" exitCode=0 Sep 14 02:52:48.308243 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.307561 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c5693c3-3949-4015-9a7b-f5451574e59c","Type":"ContainerDied","Data":"1f9cf377f97d7d63a451723694fc1b1cf331aa894737c3848b7114a399e62c3b"} Sep 14 02:52:48.310338 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.310302 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" event={"ID":"049d80ec-0298-4f01-a744-5bfcd8ea6bcf","Type":"ContainerStarted","Data":"139726dc3a7f82fd371288362e0a26c60681cc9ae91da565416e865cc792d8f0"} Sep 14 02:52:48.310445 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.310354 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" event={"ID":"049d80ec-0298-4f01-a744-5bfcd8ea6bcf","Type":"ContainerStarted","Data":"f1ab9a9bea746e6745c8c7baa9800478863156cdfd3e6be4adcdce5c8853196e"} Sep 14 02:52:48.312020 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.311990 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-688bd89fc-t949d" event={"ID":"c02c878c-a03b-42ed-b288-6b044226cee0","Type":"ContainerStarted","Data":"431c066ad78f167db37ecf331d39a69c90bb4efe1bb24ac34c9a525660f80d7b"} Sep 14 02:52:48.314203 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.314181 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v8xq9" event={"ID":"42ab45b2-5436-4392-b8a1-000f53af76df","Type":"ContainerStarted","Data":"3581fb3d13df493cf448819fbb5fe189d9699ffff10a569e134ec7e9dadfbd47"} Sep 14 02:52:48.314294 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.314202 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:52:48.314294 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.314221 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-v8xq9" event={"ID":"42ab45b2-5436-4392-b8a1-000f53af76df","Type":"ContainerStarted","Data":"948e1c02d55b99ea6a64cc12679a5961eb4ce98599a52322e1cf361ef9e4b2d4"} Sep 14 02:52:48.341643 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.341621 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:52:48.345942 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.345698 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:48.368653 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.364965 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.368653 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.365040 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.368653 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.365098 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm7zd\" (UniqueName: \"kubernetes.io/projected/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-kube-api-access-mm7zd\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.368653 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.365128 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-secret-telemeter-client\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.368653 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.365155 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-serving-certs-ca-bundle\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.368653 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.365186 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-telemeter-client-tls\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.368653 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.365208 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-federate-client-tls\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.368653 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.365348 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-metrics-client-ca\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.390031 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.389982 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-v8xq9" podStartSLOduration=5.31275423 podStartE2EDuration="6.389961779s" totalImagesPullingTime="991.145043ms" totalInitContainerRuntime="86.062506ms" isStatefulPod=true podCreationTimestamp="2026-09-14 02:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:48.389359088 +0000 UTC m=+140.119018205" watchObservedRunningTime="2026-09-14 02:52:48.389961779 +0000 UTC m=+140.119620896" Sep 14 02:52:48.390505 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.390451 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jrk97" podStartSLOduration=136.199899797 podStartE2EDuration="2m19.390438252s" totalImagesPullingTime="3.190538455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:48.357812804 +0000 UTC m=+140.087471919" watchObservedRunningTime="2026-09-14 02:52:48.390438252 +0000 UTC m=+140.120097369" Sep 14 02:52:48.433656 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.433628 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:48.465996 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.465963 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba5bd25d-67d4-45cf-a2fa-e5169274172b-trusted-ca\") pod \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " Sep 14 02:52:48.466126 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.466016 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba5bd25d-67d4-45cf-a2fa-e5169274172b-image-registry-private-configuration\") pod \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " Sep 14 02:52:48.466126 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.466063 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-certificates\") pod \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " Sep 14 02:52:48.466126 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.466101 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba5bd25d-67d4-45cf-a2fa-e5169274172b-installation-pull-secrets\") pod \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " Sep 14 02:52:48.466338 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.466159 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sczvk\" (UniqueName: \"kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-kube-api-access-sczvk\") pod \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " Sep 14 02:52:48.466338 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.466203 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba5bd25d-67d4-45cf-a2fa-e5169274172b-ca-trust-extracted\") pod \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " Sep 14 02:52:48.466338 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.466263 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-bound-sa-token\") pod \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\" (UID: \"ba5bd25d-67d4-45cf-a2fa-e5169274172b\") " Sep 14 02:52:48.466502 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.466380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.466502 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.466419 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm7zd\" (UniqueName: \"kubernetes.io/projected/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-kube-api-access-mm7zd\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.466502 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.466470 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-secret-telemeter-client\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.466645 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.466516 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-serving-certs-ca-bundle\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.466645 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.466554 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-telemeter-client-tls\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.466645 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.466580 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-federate-client-tls\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.466785 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.466701 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-metrics-client-ca\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.466834 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.466824 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.467757 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.467727 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.468313 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.468290 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-metrics-client-ca\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.469426 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.469398 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5bd25d-67d4-45cf-a2fa-e5169274172b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ba5bd25d-67d4-45cf-a2fa-e5169274172b" (UID: "ba5bd25d-67d4-45cf-a2fa-e5169274172b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 02:52:48.470558 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.469974 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ba5bd25d-67d4-45cf-a2fa-e5169274172b" (UID: "ba5bd25d-67d4-45cf-a2fa-e5169274172b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:52:48.470558 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.470254 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5bd25d-67d4-45cf-a2fa-e5169274172b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ba5bd25d-67d4-45cf-a2fa-e5169274172b" (UID: "ba5bd25d-67d4-45cf-a2fa-e5169274172b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:52:48.471306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.471283 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-serving-certs-ca-bundle\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.472126 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.472063 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-telemeter-client-tls\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.474982 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.474151 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-federate-client-tls\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.474982 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.474266 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-kube-api-access-sczvk" (OuterVolumeSpecName: "kube-api-access-sczvk") pod "ba5bd25d-67d4-45cf-a2fa-e5169274172b" (UID: "ba5bd25d-67d4-45cf-a2fa-e5169274172b"). InnerVolumeSpecName "kube-api-access-sczvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:52:48.474982 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.474624 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba5bd25d-67d4-45cf-a2fa-e5169274172b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ba5bd25d-67d4-45cf-a2fa-e5169274172b" (UID: "ba5bd25d-67d4-45cf-a2fa-e5169274172b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 02:52:48.474982 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.474661 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ba5bd25d-67d4-45cf-a2fa-e5169274172b" (UID: "ba5bd25d-67d4-45cf-a2fa-e5169274172b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:52:48.476127 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.476097 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.476543 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.476468 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-secret-telemeter-client\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.484906 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.484883 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm7zd\" (UniqueName: \"kubernetes.io/projected/1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b-kube-api-access-mm7zd\") pod \"telemeter-client-56b4cc6586-zhfrt\" (UID: \"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b\") " pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.491216 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.491186 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba5bd25d-67d4-45cf-a2fa-e5169274172b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ba5bd25d-67d4-45cf-a2fa-e5169274172b" (UID: "ba5bd25d-67d4-45cf-a2fa-e5169274172b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 02:52:48.510326 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.510297 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:48.567905 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.567863 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sczvk\" (UniqueName: \"kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-kube-api-access-sczvk\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 02:52:48.567905 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.567896 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba5bd25d-67d4-45cf-a2fa-e5169274172b-ca-trust-extracted\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 02:52:48.567905 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.567906 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-bound-sa-token\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 02:52:48.567905 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.567914 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba5bd25d-67d4-45cf-a2fa-e5169274172b-trusted-ca\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 02:52:48.568166 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.567923 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba5bd25d-67d4-45cf-a2fa-e5169274172b-image-registry-private-configuration\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 02:52:48.568166 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.567931 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-certificates\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 02:52:48.568166 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.567943 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba5bd25d-67d4-45cf-a2fa-e5169274172b-installation-pull-secrets\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 02:52:48.584855 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.584826 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" Sep 14 02:52:48.609471 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.609433 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:48.893465 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:48.893434 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56b4cc6586-zhfrt"] Sep 14 02:52:48.895511 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:52:48.895461 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c2ae1ea_bde9_4ff8_8f4c_0e28e39ff11b.slice/crio-0e037596a625659cb559f04fcfb3f3929e04d56df1eb2c0906c77864fc0d131c WatchSource:0}: Error finding container 0e037596a625659cb559f04fcfb3f3929e04d56df1eb2c0906c77864fc0d131c: Status 404 returned error can't find the container with id 0e037596a625659cb559f04fcfb3f3929e04d56df1eb2c0906c77864fc0d131c Sep 14 02:52:49.062271 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.062189 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:49.067212 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.066413 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 02:52:49.080946 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.080920 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.084319 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.084294 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 14 02:52:49.084867 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.084846 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 14 02:52:49.085229 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.085211 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bdj15i2kn1f8h\"" Sep 14 02:52:49.085439 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.085418 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 14 02:52:49.086890 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.086872 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 14 02:52:49.087263 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.087247 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 14 02:52:49.087599 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.087584 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rgz2n\"" Sep 14 02:52:49.087898 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.087885 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 14 02:52:49.088116 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.088103 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 14 02:52:49.099138 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.088624 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 14 02:52:49.099394 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.088632 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 14 02:52:49.099621 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.088712 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 14 02:52:49.099726 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.088763 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 14 02:52:49.099813 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.088806 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 14 02:52:49.099954 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.088929 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 14 02:52:49.100045 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.090567 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 14 02:52:49.100133 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.096423 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 02:52:49.179505 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.179454 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0bfc948-e0b8-47d2-9e84-95f519f3411a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.180277 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.179527 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0bfc948-e0b8-47d2-9e84-95f519f3411a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.180277 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.179578 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.180277 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.179606 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0bfc948-e0b8-47d2-9e84-95f519f3411a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.180277 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.179632 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.180277 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.179671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0bfc948-e0b8-47d2-9e84-95f519f3411a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.180277 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.179694 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.180277 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.179723 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.180277 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.179745 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.180277 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.179776 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.180277 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.179818 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0bfc948-e0b8-47d2-9e84-95f519f3411a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.180277 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.179850 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0bfc948-e0b8-47d2-9e84-95f519f3411a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.180277 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.179892 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8zr9\" (UniqueName: \"kubernetes.io/projected/e0bfc948-e0b8-47d2-9e84-95f519f3411a-kube-api-access-b8zr9\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.180277 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.179923 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-config\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.180277 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.179947 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0bfc948-e0b8-47d2-9e84-95f519f3411a-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.180277 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.179969 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.180277 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.180011 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0bfc948-e0b8-47d2-9e84-95f519f3411a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.181021 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.180079 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0bfc948-e0b8-47d2-9e84-95f519f3411a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.181021 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.180114 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0bfc948-e0b8-47d2-9e84-95f519f3411a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.181021 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.180141 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.281161 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.281122 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8zr9\" (UniqueName: \"kubernetes.io/projected/e0bfc948-e0b8-47d2-9e84-95f519f3411a-kube-api-access-b8zr9\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.281673 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.281178 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-config\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.281673 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.281199 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0bfc948-e0b8-47d2-9e84-95f519f3411a-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.281673 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.281220 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.282043 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.282015 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0bfc948-e0b8-47d2-9e84-95f519f3411a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.282144 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.282068 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0bfc948-e0b8-47d2-9e84-95f519f3411a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.282144 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.282094 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0bfc948-e0b8-47d2-9e84-95f519f3411a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.282144 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.282128 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.282292 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.282182 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0bfc948-e0b8-47d2-9e84-95f519f3411a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.282292 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.282217 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0bfc948-e0b8-47d2-9e84-95f519f3411a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.282292 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.282264 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.282632 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.282292 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0bfc948-e0b8-47d2-9e84-95f519f3411a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.282632 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.282317 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.282632 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.282394 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0bfc948-e0b8-47d2-9e84-95f519f3411a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.282632 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.282419 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.282632 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.282451 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.282632 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.282491 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.282632 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.282523 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.282632 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.282571 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0bfc948-e0b8-47d2-9e84-95f519f3411a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.282632 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.282612 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0bfc948-e0b8-47d2-9e84-95f519f3411a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.283499 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.283381 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0bfc948-e0b8-47d2-9e84-95f519f3411a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.283975 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.283881 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0bfc948-e0b8-47d2-9e84-95f519f3411a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.284158 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.284136 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0bfc948-e0b8-47d2-9e84-95f519f3411a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.284908 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.284713 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.285640 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.285616 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0bfc948-e0b8-47d2-9e84-95f519f3411a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.287636 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.286664 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-config\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.287636 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.286799 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0bfc948-e0b8-47d2-9e84-95f519f3411a-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.287636 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.287355 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0bfc948-e0b8-47d2-9e84-95f519f3411a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.290766 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.290619 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0bfc948-e0b8-47d2-9e84-95f519f3411a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.291851 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.291822 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.291956 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.291923 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.292437 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.292394 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0bfc948-e0b8-47d2-9e84-95f519f3411a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.294145 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.293605 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.296677 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.296645 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0bfc948-e0b8-47d2-9e84-95f519f3411a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.300758 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.300701 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.301537 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.301189 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0bfc948-e0b8-47d2-9e84-95f519f3411a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.301631 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.301599 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.301886 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.301869 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.301970 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.301950 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8zr9\" (UniqueName: \"kubernetes.io/projected/e0bfc948-e0b8-47d2-9e84-95f519f3411a-kube-api-access-b8zr9\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.302889 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.302292 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0bfc948-e0b8-47d2-9e84-95f519f3411a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0bfc948-e0b8-47d2-9e84-95f519f3411a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.324204 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.324088 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" event={"ID":"049d80ec-0298-4f01-a744-5bfcd8ea6bcf","Type":"ContainerStarted","Data":"32c4014ba97b02b3e4275d5bab485b7ce8e59985bb2d69ff24617043e77d9ed7"} Sep 14 02:52:49.326250 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.325693 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" event={"ID":"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b","Type":"ContainerStarted","Data":"0e037596a625659cb559f04fcfb3f3929e04d56df1eb2c0906c77864fc0d131c"} Sep 14 02:52:49.326968 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.326417 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76c79f4bcc-mg4xz" Sep 14 02:52:49.369041 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.368989 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-w24sd" podStartSLOduration=4.200524625 podStartE2EDuration="7.368971437s" totalImagesPullingTime="3.168446812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:49.345167173 +0000 UTC m=+141.074826288" watchObservedRunningTime="2026-09-14 02:52:49.368971437 +0000 UTC m=+141.098630555" Sep 14 02:52:49.376532 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.376430 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76c79f4bcc-mg4xz"] Sep 14 02:52:49.385448 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.382104 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-76c79f4bcc-mg4xz"] Sep 14 02:52:49.400241 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.400215 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:52:49.488598 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.486194 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba5bd25d-67d4-45cf-a2fa-e5169274172b-registry-tls\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 02:52:49.774009 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:49.773907 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 14 02:52:50.344200 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:52:50.344088 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0bfc948_e0b8_47d2_9e84_95f519f3411a.slice/crio-4f5492820494242ab8a474b11b130bfb764b750a126b7bea9693a1c37d5bd895 WatchSource:0}: Error finding container 4f5492820494242ab8a474b11b130bfb764b750a126b7bea9693a1c37d5bd895: Status 404 returned error can't find the container with id 4f5492820494242ab8a474b11b130bfb764b750a126b7bea9693a1c37d5bd895 Sep 14 02:52:50.750160 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:50.750108 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba5bd25d-67d4-45cf-a2fa-e5169274172b" path="/var/lib/kubelet/pods/ba5bd25d-67d4-45cf-a2fa-e5169274172b/volumes" Sep 14 02:52:51.337856 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:51.337811 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0bfc948-e0b8-47d2-9e84-95f519f3411a","Type":"ContainerStarted","Data":"4f5492820494242ab8a474b11b130bfb764b750a126b7bea9693a1c37d5bd895"} Sep 14 02:52:52.347658 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:52.347617 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0bfc948-e0b8-47d2-9e84-95f519f3411a","Type":"ContainerStarted","Data":"61f98704133f1a113d24d207fd6a8ad68e86188094ecaf2490b3a98a5c72f3b8"} Sep 14 02:52:52.418516 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:52.418401 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:52.516575 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:52.516544 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:53.268162 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:53.268125 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:53.327708 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:53.327676 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:53.355681 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:53.355649 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-688bd89fc-t949d" event={"ID":"c02c878c-a03b-42ed-b288-6b044226cee0","Type":"ContainerStarted","Data":"219a3ace6f3276ec633988d00494bf4b3fd0042f60b49449ed350f29b7cc84d1"} Sep 14 02:52:53.357731 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:53.357698 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" event={"ID":"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b","Type":"ContainerStarted","Data":"b2132b12147f19a24acb6c6368d25f918d9c74d008aaeb021b93316cf6f89084"} Sep 14 02:52:53.357855 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:53.357751 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" event={"ID":"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b","Type":"ContainerStarted","Data":"e8ff8d1507d18dc5718b53fe630e19a0e5d0727e8e58e78cdb8eaa33e65345c4"} Sep 14 02:52:53.357855 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:53.357779 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" event={"ID":"1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b","Type":"ContainerStarted","Data":"a167d796d0b131212c7815a88a2f5231e8e276e0fa8eb05bb4faccc419ec971c"} Sep 14 02:52:53.359281 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:53.359259 2564 generic.go:356] "Generic (PLEG): container finished" podID="e0bfc948-e0b8-47d2-9e84-95f519f3411a" containerID="61f98704133f1a113d24d207fd6a8ad68e86188094ecaf2490b3a98a5c72f3b8" exitCode=0 Sep 14 02:52:53.359380 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:53.359308 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0bfc948-e0b8-47d2-9e84-95f519f3411a","Type":"ContainerDied","Data":"61f98704133f1a113d24d207fd6a8ad68e86188094ecaf2490b3a98a5c72f3b8"} Sep 14 02:52:53.362577 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:53.362551 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c5693c3-3949-4015-9a7b-f5451574e59c","Type":"ContainerStarted","Data":"82a9949041eabac270ace877911d93d6217c3e881b9803d337f61759f829dd5b"} Sep 14 02:52:53.362659 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:53.362608 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c5693c3-3949-4015-9a7b-f5451574e59c","Type":"ContainerStarted","Data":"f0707bf847357ff40a6c4f80f112f008ffc85b43eb8593159f64ce464fcc76ab"} Sep 14 02:52:53.362659 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:53.362640 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c5693c3-3949-4015-9a7b-f5451574e59c","Type":"ContainerStarted","Data":"5801f8f3bc1529dcaae014dbdcd5babdac6e5346ebc24eda9e13aeb51aad9b53"} Sep 14 02:52:53.362724 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:53.362669 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c5693c3-3949-4015-9a7b-f5451574e59c","Type":"ContainerStarted","Data":"abce06c42056fee9c3c00f837d7b6529f3fdfce38a793dc65136a4fe51ac5db1"} Sep 14 02:52:53.362724 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:53.362697 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c5693c3-3949-4015-9a7b-f5451574e59c","Type":"ContainerStarted","Data":"c2521b1aca790dbf1f75fe18fd8ee11bb4dd34ba5c5fcf8c45d916d1149d37d5"} Sep 14 02:52:53.373720 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:53.373677 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-688bd89fc-t949d" podStartSLOduration=2.31922665 podStartE2EDuration="6.373662108s" totalImagesPullingTime="4.054435458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:53.371905364 +0000 UTC m=+145.101564480" watchObservedRunningTime="2026-09-14 02:52:53.373662108 +0000 UTC m=+145.103321224" Sep 14 02:52:53.395668 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:53.395609 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-56b4cc6586-zhfrt" podStartSLOduration=2.072599534 podStartE2EDuration="5.395592869s" totalImagesPullingTime="3.322993335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:53.393609576 +0000 UTC m=+145.123268691" watchObservedRunningTime="2026-09-14 02:52:53.395592869 +0000 UTC m=+145.125251984" Sep 14 02:52:54.142526 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:54.142504 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:52:54.244920 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:54.244892 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:54.351551 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:54.351500 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:52:54.371806 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:54.371769 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2c5693c3-3949-4015-9a7b-f5451574e59c","Type":"ContainerStarted","Data":"0e92f469aaf0436adc4c9f34771c8da6c13496f45462925339c7c00db0fb9367"} Sep 14 02:52:54.406918 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:54.406869 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.134369486 podStartE2EDuration="10.406850696s" totalImagesPullingTime="8.244378739s" totalInitContainerRuntime="28.102471ms" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:52:54.405655389 +0000 UTC m=+146.135314505" watchObservedRunningTime="2026-09-14 02:52:54.406850696 +0000 UTC m=+146.136509813" Sep 14 02:52:57.389262 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:57.389226 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0bfc948-e0b8-47d2-9e84-95f519f3411a","Type":"ContainerStarted","Data":"ca0fa2eae59cc9e121c994a0e37db027e76c2ca7d21cba97cadf15557b7d619a"} Sep 14 02:52:57.389729 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:52:57.389324 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0bfc948-e0b8-47d2-9e84-95f519f3411a","Type":"ContainerStarted","Data":"75be9344045787a41b4c783ce398b7d9b9f635c36ee4b5a809327e3d590706a5"} Sep 14 02:53:00.260001 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:00.259964 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:00.314822 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:00.314792 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:00.404226 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:00.404195 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0bfc948-e0b8-47d2-9e84-95f519f3411a","Type":"ContainerStarted","Data":"f3138b6059cdde44b448c234ffba91497853f3cd8461b273248df289d7514ab7"} Sep 14 02:53:00.404374 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:00.404271 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0bfc948-e0b8-47d2-9e84-95f519f3411a","Type":"ContainerStarted","Data":"848b9b5dd15c58db5b91a114814611b756bdc1c10b65971cebb808717f9e4afb"} Sep 14 02:53:00.404374 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:00.404309 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0bfc948-e0b8-47d2-9e84-95f519f3411a","Type":"ContainerStarted","Data":"b47a001fe882c8f5fac00897e0a631587644d4ace94320c1ec73a052bc69b2f6"} Sep 14 02:53:00.404374 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:00.404348 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0bfc948-e0b8-47d2-9e84-95f519f3411a","Type":"ContainerStarted","Data":"29a235a31cbfca076bcf366a71024d8c4fbcdd94f3d164b6b7110f3a9d8ee4e4"} Sep 14 02:53:00.431133 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:00.431089 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.498726328 podStartE2EDuration="11.431075146s" totalImagesPullingTime="5.901389984s" totalInitContainerRuntime="30.958834ms" isStatefulPod=false podCreationTimestamp="2026-09-14 02:52:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:00.429666647 +0000 UTC m=+152.159325761" watchObservedRunningTime="2026-09-14 02:53:00.431075146 +0000 UTC m=+152.160734260" Sep 14 02:53:02.302051 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:02.302021 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:02.402572 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:02.402532 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:03.458414 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:03.458381 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:03.567034 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:03.566990 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:03.659989 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:03.659963 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:04.220683 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:04.220647 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:04.312207 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:04.312173 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:04.401371 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:04.401341 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:53:05.531809 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:05.531772 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:05.633869 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:05.633842 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:06.154618 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:06.154588 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:06.253537 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:06.253502 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:06.282461 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:53:06.282433 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-znkd5" podUID="47b16be3-755a-4a89-a2ef-830c72498f05" Sep 14 02:53:06.425527 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:06.425459 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-znkd5" Sep 14 02:53:07.610971 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:07.610942 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:53:07.611318 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:07.610995 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:53:09.347877 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:09.347843 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:09.430163 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:09.430130 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:10.987184 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:10.987146 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:11.166672 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:11.166645 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls\") pod \"dns-default-znkd5\" (UID: \"47b16be3-755a-4a89-a2ef-830c72498f05\") " pod="openshift-dns/dns-default-znkd5" Sep 14 02:53:11.168986 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:11.168966 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47b16be3-755a-4a89-a2ef-830c72498f05-metrics-tls\") pod \"dns-default-znkd5\" (UID: \"47b16be3-755a-4a89-a2ef-830c72498f05\") " pod="openshift-dns/dns-default-znkd5" Sep 14 02:53:11.194709 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:11.194669 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:11.229921 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:11.229901 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nzbvq\"" Sep 14 02:53:11.236964 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:11.236948 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-znkd5" Sep 14 02:53:11.309935 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:11.309907 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:11.544693 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:11.544671 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-znkd5"] Sep 14 02:53:11.547098 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:53:11.547064 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b16be3_755a_4a89_a2ef_830c72498f05.slice/crio-b7767ee2dee83824afbcfd7f019e641287fbeefe968e73095356c7c6f5b2a9fb WatchSource:0}: Error finding container b7767ee2dee83824afbcfd7f019e641287fbeefe968e73095356c7c6f5b2a9fb: Status 404 returned error can't find the container with id b7767ee2dee83824afbcfd7f019e641287fbeefe968e73095356c7c6f5b2a9fb Sep 14 02:53:12.315295 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.315237 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c9d6f945d-ph585" podUID="d5c09dff-03ba-4fac-ae74-a41b1812bf38" containerName="console" containerID="cri-o://cfdcaa38aec9653c1c83b0b7c04a1347df1303ebac5643c4c9fb8daded4d0a8f" gracePeriod=15 Sep 14 02:53:12.449612 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.449582 2564 generic.go:356] "Generic (PLEG): container finished" podID="d5c09dff-03ba-4fac-ae74-a41b1812bf38" containerID="cfdcaa38aec9653c1c83b0b7c04a1347df1303ebac5643c4c9fb8daded4d0a8f" exitCode=0 Sep 14 02:53:12.449737 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.449656 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9d6f945d-ph585" event={"ID":"d5c09dff-03ba-4fac-ae74-a41b1812bf38","Type":"ContainerDied","Data":"cfdcaa38aec9653c1c83b0b7c04a1347df1303ebac5643c4c9fb8daded4d0a8f"} Sep 14 02:53:12.450980 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.450955 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-znkd5" event={"ID":"47b16be3-755a-4a89-a2ef-830c72498f05","Type":"ContainerStarted","Data":"b7767ee2dee83824afbcfd7f019e641287fbeefe968e73095356c7c6f5b2a9fb"} Sep 14 02:53:12.576364 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.576306 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:53:12.678583 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.678552 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5c09dff-03ba-4fac-ae74-a41b1812bf38-console-config\") pod \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " Sep 14 02:53:12.678734 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.678608 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c09dff-03ba-4fac-ae74-a41b1812bf38-service-ca\") pod \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " Sep 14 02:53:12.678734 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.678633 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5c09dff-03ba-4fac-ae74-a41b1812bf38-oauth-serving-cert\") pod \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " Sep 14 02:53:12.678734 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.678705 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5c09dff-03ba-4fac-ae74-a41b1812bf38-console-oauth-config\") pod \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " Sep 14 02:53:12.678734 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.678721 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwtwm\" (UniqueName: \"kubernetes.io/projected/d5c09dff-03ba-4fac-ae74-a41b1812bf38-kube-api-access-xwtwm\") pod \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " Sep 14 02:53:12.678951 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.678745 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c09dff-03ba-4fac-ae74-a41b1812bf38-console-serving-cert\") pod \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\" (UID: \"d5c09dff-03ba-4fac-ae74-a41b1812bf38\") " Sep 14 02:53:12.679006 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.678978 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c09dff-03ba-4fac-ae74-a41b1812bf38-service-ca" (OuterVolumeSpecName: "service-ca") pod "d5c09dff-03ba-4fac-ae74-a41b1812bf38" (UID: "d5c09dff-03ba-4fac-ae74-a41b1812bf38"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:53:12.679058 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.678994 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c09dff-03ba-4fac-ae74-a41b1812bf38-console-config" (OuterVolumeSpecName: "console-config") pod "d5c09dff-03ba-4fac-ae74-a41b1812bf38" (UID: "d5c09dff-03ba-4fac-ae74-a41b1812bf38"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:53:12.679111 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.679058 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c09dff-03ba-4fac-ae74-a41b1812bf38-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d5c09dff-03ba-4fac-ae74-a41b1812bf38" (UID: "d5c09dff-03ba-4fac-ae74-a41b1812bf38"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 02:53:12.681262 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.681224 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c09dff-03ba-4fac-ae74-a41b1812bf38-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d5c09dff-03ba-4fac-ae74-a41b1812bf38" (UID: "d5c09dff-03ba-4fac-ae74-a41b1812bf38"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 02:53:12.681390 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.681263 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c09dff-03ba-4fac-ae74-a41b1812bf38-kube-api-access-xwtwm" (OuterVolumeSpecName: "kube-api-access-xwtwm") pod "d5c09dff-03ba-4fac-ae74-a41b1812bf38" (UID: "d5c09dff-03ba-4fac-ae74-a41b1812bf38"). InnerVolumeSpecName "kube-api-access-xwtwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:53:12.681390 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.681354 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c09dff-03ba-4fac-ae74-a41b1812bf38-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d5c09dff-03ba-4fac-ae74-a41b1812bf38" (UID: "d5c09dff-03ba-4fac-ae74-a41b1812bf38"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 02:53:12.779786 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.779758 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5c09dff-03ba-4fac-ae74-a41b1812bf38-console-oauth-config\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 02:53:12.779786 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.779788 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwtwm\" (UniqueName: \"kubernetes.io/projected/d5c09dff-03ba-4fac-ae74-a41b1812bf38-kube-api-access-xwtwm\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 02:53:12.779982 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.779797 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c09dff-03ba-4fac-ae74-a41b1812bf38-console-serving-cert\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 02:53:12.779982 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.779805 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5c09dff-03ba-4fac-ae74-a41b1812bf38-console-config\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 02:53:12.779982 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.779813 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c09dff-03ba-4fac-ae74-a41b1812bf38-service-ca\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 02:53:12.779982 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:12.779820 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5c09dff-03ba-4fac-ae74-a41b1812bf38-oauth-serving-cert\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 02:53:13.455915 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:13.455886 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9d6f945d-ph585" Sep 14 02:53:13.456344 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:13.455880 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9d6f945d-ph585" event={"ID":"d5c09dff-03ba-4fac-ae74-a41b1812bf38","Type":"ContainerDied","Data":"f6f5d060087f3a9089401a09720804e0436c8485b0559ca1bb63dfc80309b9c9"} Sep 14 02:53:13.456344 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:13.456054 2564 scope.go:117] "RemoveContainer" containerID="cfdcaa38aec9653c1c83b0b7c04a1347df1303ebac5643c4c9fb8daded4d0a8f" Sep 14 02:53:13.475443 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:13.475415 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c9d6f945d-ph585"] Sep 14 02:53:13.477768 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:13.477750 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c9d6f945d-ph585"] Sep 14 02:53:14.461880 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:14.461850 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-znkd5" event={"ID":"47b16be3-755a-4a89-a2ef-830c72498f05","Type":"ContainerStarted","Data":"65efc212316022e06bccde0bfc7413b732e08054a4a2e0deb077d1168469293a"} Sep 14 02:53:14.462222 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:14.461896 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-znkd5" event={"ID":"47b16be3-755a-4a89-a2ef-830c72498f05","Type":"ContainerStarted","Data":"d16938dfdff9f635fa486f7a5d43379ae8bdaded330de60d6c59a94e848e1316"} Sep 14 02:53:14.462222 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:14.461926 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-znkd5" Sep 14 02:53:14.479513 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:14.479452 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-znkd5" podStartSLOduration=129.475119602 podStartE2EDuration="2m11.479438832s" totalImagesPullingTime="2.00431923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:53:14.47930138 +0000 UTC m=+166.208960495" watchObservedRunningTime="2026-09-14 02:53:14.479438832 +0000 UTC m=+166.209097947" Sep 14 02:53:14.748281 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:14.748221 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5c09dff-03ba-4fac-ae74-a41b1812bf38" path="/var/lib/kubelet/pods/d5c09dff-03ba-4fac-ae74-a41b1812bf38/volumes" Sep 14 02:53:18.552567 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:18.552508 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:18.615423 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:18.615365 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:24.474082 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:24.474012 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-znkd5" Sep 14 02:53:27.616170 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:27.616141 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:53:27.619798 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:27.619777 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-688bd89fc-t949d" Sep 14 02:53:33.601057 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:33.601023 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:33.699931 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:33.699901 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:49.400797 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:49.400762 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:53:49.421671 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:49.421639 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:53:49.620262 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:49.620232 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 14 02:53:52.667181 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:52.667147 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:58.994606 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:58.994574 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:53:59.088964 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:53:59.088927 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:54:06.182760 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:54:06.182711 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:54:06.275704 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:54:06.275666 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:54:10.965784 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:54:10.965736 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:54:11.067120 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:54:11.067083 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:54:16.748852 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:54:16.748810 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:54:16.851447 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:54:16.851410 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:54:16.951564 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:54:16.951533 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:54:17.052806 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:54:17.052717 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:54:18.585916 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:54:18.585880 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:54:18.716019 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:54:18.715980 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:54:58.594041 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:54:58.594009 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:54:58.695412 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:54:58.695378 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:55:00.403589 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:00.403550 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:55:06.637712 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:06.637677 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:55:09.017852 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:09.017818 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:55:09.193197 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:09.193166 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:55:20.961831 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:20.961791 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:55:21.346239 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:21.346144 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:55:25.174062 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:25.174015 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:55:25.274572 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:25.274540 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:55:27.982853 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:27.982813 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:55:28.213564 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:28.213531 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:55:28.594002 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:28.593970 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:55:28.700213 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:28.700180 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bwv6_4369115a-4032-4f5a-a3e4-c02efe665617/ovn-acl-logging/0.log" Sep 14 02:55:28.703682 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:28.703664 2564 kubelet.go:1648] "Image garbage collection succeeded" Sep 14 02:55:28.705431 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:28.705413 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bwv6_4369115a-4032-4f5a-a3e4-c02efe665617/ovn-acl-logging/0.log" Sep 14 02:55:28.771965 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:28.771838 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:55:30.840176 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:30.840138 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:55:31.012913 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:31.012886 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jcjw6"] Sep 14 02:55:31.013211 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:31.013199 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c09dff-03ba-4fac-ae74-a41b1812bf38" containerName="console" Sep 14 02:55:31.013260 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:31.013213 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c09dff-03ba-4fac-ae74-a41b1812bf38" containerName="console" Sep 14 02:55:31.013294 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:31.013262 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c09dff-03ba-4fac-ae74-a41b1812bf38" containerName="console" Sep 14 02:55:31.015794 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:31.015777 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcjw6" Sep 14 02:55:31.019872 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:31.019850 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 14 02:55:31.020612 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:31.020557 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 14 02:55:31.037383 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:31.037359 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jcjw6"] Sep 14 02:55:31.064156 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:31.064121 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:55:31.110069 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:31.109998 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d22ceb5-bf2f-477a-9dcd-e4c58ae71d35-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jcjw6\" (UID: \"9d22ceb5-bf2f-477a-9dcd-e4c58ae71d35\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcjw6" Sep 14 02:55:31.110069 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:31.110059 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tspcp\" (UniqueName: \"kubernetes.io/projected/9d22ceb5-bf2f-477a-9dcd-e4c58ae71d35-kube-api-access-tspcp\") pod \"cert-manager-cainjector-559cdd8c87-jcjw6\" (UID: \"9d22ceb5-bf2f-477a-9dcd-e4c58ae71d35\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcjw6" Sep 14 02:55:31.210824 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:31.210790 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d22ceb5-bf2f-477a-9dcd-e4c58ae71d35-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jcjw6\" (UID: \"9d22ceb5-bf2f-477a-9dcd-e4c58ae71d35\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcjw6" Sep 14 02:55:31.210936 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:31.210847 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tspcp\" (UniqueName: \"kubernetes.io/projected/9d22ceb5-bf2f-477a-9dcd-e4c58ae71d35-kube-api-access-tspcp\") pod \"cert-manager-cainjector-559cdd8c87-jcjw6\" (UID: \"9d22ceb5-bf2f-477a-9dcd-e4c58ae71d35\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcjw6" Sep 14 02:55:31.220952 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:31.220921 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d22ceb5-bf2f-477a-9dcd-e4c58ae71d35-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jcjw6\" (UID: \"9d22ceb5-bf2f-477a-9dcd-e4c58ae71d35\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcjw6" Sep 14 02:55:31.221278 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:31.221258 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tspcp\" (UniqueName: \"kubernetes.io/projected/9d22ceb5-bf2f-477a-9dcd-e4c58ae71d35-kube-api-access-tspcp\") pod \"cert-manager-cainjector-559cdd8c87-jcjw6\" (UID: \"9d22ceb5-bf2f-477a-9dcd-e4c58ae71d35\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcjw6" Sep 14 02:55:31.325269 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:31.325235 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcjw6" Sep 14 02:55:31.605338 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:31.605196 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jcjw6"] Sep 14 02:55:31.608105 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:55:31.608074 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d22ceb5_bf2f_477a_9dcd_e4c58ae71d35.slice/crio-5ecf945424b0c6c46a447263e63e696d02eea7971322209fa3299d5c2a035d75 WatchSource:0}: Error finding container 5ecf945424b0c6c46a447263e63e696d02eea7971322209fa3299d5c2a035d75: Status 404 returned error can't find the container with id 5ecf945424b0c6c46a447263e63e696d02eea7971322209fa3299d5c2a035d75 Sep 14 02:55:31.610012 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:31.609996 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 02:55:32.000898 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:32.000861 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcjw6" event={"ID":"9d22ceb5-bf2f-477a-9dcd-e4c58ae71d35","Type":"ContainerStarted","Data":"5ecf945424b0c6c46a447263e63e696d02eea7971322209fa3299d5c2a035d75"} Sep 14 02:55:34.276191 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:34.276149 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:55:34.359061 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:34.359004 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:55:35.018142 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:35.018049 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcjw6" event={"ID":"9d22ceb5-bf2f-477a-9dcd-e4c58ae71d35","Type":"ContainerStarted","Data":"ddca5ee5bf9d7e6381bf9c89998a9ca5bfac4f61e3b02cd284554c2c03a26293"} Sep 14 02:55:38.080170 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:38.080134 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:55:38.609408 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:38.609374 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:55:38.709865 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:38.709830 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:55:40.617528 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:40.617453 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcjw6" podStartSLOduration=7.476062978 podStartE2EDuration="10.617436491s" totalImagesPullingTime="3.141373513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:55:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:55:35.041762495 +0000 UTC m=+306.771421612" watchObservedRunningTime="2026-09-14 02:55:40.617436491 +0000 UTC m=+312.347095629" Sep 14 02:55:40.618628 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:40.618607 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l5sdn"] Sep 14 02:55:40.621858 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:40.621843 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l5sdn" Sep 14 02:55:40.624524 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:40.624496 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 14 02:55:40.625167 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:40.625149 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 14 02:55:40.625167 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:40.625162 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-prfb2\"" Sep 14 02:55:40.633882 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:40.633862 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l5sdn"] Sep 14 02:55:40.789199 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:40.789167 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc9f26cd-12ca-42a4-a680-93e4b1878b86-tmp\") pod \"openshift-lws-operator-fd8ccff4c-l5sdn\" (UID: \"fc9f26cd-12ca-42a4-a680-93e4b1878b86\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l5sdn" Sep 14 02:55:40.789363 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:40.789211 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-867s9\" (UniqueName: \"kubernetes.io/projected/fc9f26cd-12ca-42a4-a680-93e4b1878b86-kube-api-access-867s9\") pod \"openshift-lws-operator-fd8ccff4c-l5sdn\" (UID: \"fc9f26cd-12ca-42a4-a680-93e4b1878b86\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l5sdn" Sep 14 02:55:40.890496 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:40.890396 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc9f26cd-12ca-42a4-a680-93e4b1878b86-tmp\") pod \"openshift-lws-operator-fd8ccff4c-l5sdn\" (UID: \"fc9f26cd-12ca-42a4-a680-93e4b1878b86\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l5sdn" Sep 14 02:55:40.890496 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:40.890441 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-867s9\" (UniqueName: \"kubernetes.io/projected/fc9f26cd-12ca-42a4-a680-93e4b1878b86-kube-api-access-867s9\") pod \"openshift-lws-operator-fd8ccff4c-l5sdn\" (UID: \"fc9f26cd-12ca-42a4-a680-93e4b1878b86\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l5sdn" Sep 14 02:55:40.890857 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:40.890835 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc9f26cd-12ca-42a4-a680-93e4b1878b86-tmp\") pod \"openshift-lws-operator-fd8ccff4c-l5sdn\" (UID: \"fc9f26cd-12ca-42a4-a680-93e4b1878b86\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l5sdn" Sep 14 02:55:40.899932 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:40.899909 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-867s9\" (UniqueName: \"kubernetes.io/projected/fc9f26cd-12ca-42a4-a680-93e4b1878b86-kube-api-access-867s9\") pod \"openshift-lws-operator-fd8ccff4c-l5sdn\" (UID: \"fc9f26cd-12ca-42a4-a680-93e4b1878b86\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l5sdn" Sep 14 02:55:40.930665 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:40.930632 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l5sdn" Sep 14 02:55:41.199875 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:41.199852 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l5sdn"] Sep 14 02:55:41.202356 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:55:41.202333 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc9f26cd_12ca_42a4_a680_93e4b1878b86.slice/crio-bce6d4f19280c450cf00d5f3c5fba484e71d12632de3dfc0919e07175718a890 WatchSource:0}: Error finding container bce6d4f19280c450cf00d5f3c5fba484e71d12632de3dfc0919e07175718a890: Status 404 returned error can't find the container with id bce6d4f19280c450cf00d5f3c5fba484e71d12632de3dfc0919e07175718a890 Sep 14 02:55:42.046290 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:42.046255 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l5sdn" event={"ID":"fc9f26cd-12ca-42a4-a680-93e4b1878b86","Type":"ContainerStarted","Data":"bce6d4f19280c450cf00d5f3c5fba484e71d12632de3dfc0919e07175718a890"} Sep 14 02:55:51.535449 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:51.535415 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:55:51.636963 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:55:51.636927 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:03.134435 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:03.134399 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l5sdn" event={"ID":"fc9f26cd-12ca-42a4-a680-93e4b1878b86","Type":"ContainerStarted","Data":"b3890ce87ffdabbb0da1c4cdedfab98da74cdfcf1fbd7d2eff27720d669a35f1"} Sep 14 02:56:03.153177 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:03.153116 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l5sdn" podStartSLOduration=2.027739456 podStartE2EDuration="23.153097664s" totalImagesPullingTime="21.125358208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:56:03.151569583 +0000 UTC m=+334.881228700" watchObservedRunningTime="2026-09-14 02:56:03.153097664 +0000 UTC m=+334.882756796" Sep 14 02:56:03.983279 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:03.983243 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:04.081679 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:04.081645 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:05.642259 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:05.642223 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:11.876871 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:11.876838 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:11.953306 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:11.953275 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5647bd797c-fzbg8"] Sep 14 02:56:11.955462 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:11.955443 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5647bd797c-fzbg8" Sep 14 02:56:11.959848 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:11.959827 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 14 02:56:11.959954 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:11.959827 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 14 02:56:11.959954 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:11.959827 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 14 02:56:11.960193 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:11.960175 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2ts6x\"" Sep 14 02:56:11.969200 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:11.969179 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5647bd797c-fzbg8"] Sep 14 02:56:11.979945 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:11.979901 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:12.040179 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:12.040145 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57m6p\" (UniqueName: \"kubernetes.io/projected/3201c1d8-e6fc-43bd-8b88-4c5827d6b74d-kube-api-access-57m6p\") pod \"lws-controller-manager-5647bd797c-fzbg8\" (UID: \"3201c1d8-e6fc-43bd-8b88-4c5827d6b74d\") " pod="openshift-lws-operator/lws-controller-manager-5647bd797c-fzbg8" Sep 14 02:56:12.040377 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:12.040211 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3201c1d8-e6fc-43bd-8b88-4c5827d6b74d-metrics-cert\") pod \"lws-controller-manager-5647bd797c-fzbg8\" (UID: \"3201c1d8-e6fc-43bd-8b88-4c5827d6b74d\") " pod="openshift-lws-operator/lws-controller-manager-5647bd797c-fzbg8" Sep 14 02:56:12.040377 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:12.040323 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3201c1d8-e6fc-43bd-8b88-4c5827d6b74d-manager-config\") pod \"lws-controller-manager-5647bd797c-fzbg8\" (UID: \"3201c1d8-e6fc-43bd-8b88-4c5827d6b74d\") " pod="openshift-lws-operator/lws-controller-manager-5647bd797c-fzbg8" Sep 14 02:56:12.040513 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:12.040405 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3201c1d8-e6fc-43bd-8b88-4c5827d6b74d-cert\") pod \"lws-controller-manager-5647bd797c-fzbg8\" (UID: \"3201c1d8-e6fc-43bd-8b88-4c5827d6b74d\") " pod="openshift-lws-operator/lws-controller-manager-5647bd797c-fzbg8" Sep 14 02:56:12.141277 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:12.141198 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57m6p\" (UniqueName: \"kubernetes.io/projected/3201c1d8-e6fc-43bd-8b88-4c5827d6b74d-kube-api-access-57m6p\") pod \"lws-controller-manager-5647bd797c-fzbg8\" (UID: \"3201c1d8-e6fc-43bd-8b88-4c5827d6b74d\") " pod="openshift-lws-operator/lws-controller-manager-5647bd797c-fzbg8" Sep 14 02:56:12.141277 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:12.141244 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3201c1d8-e6fc-43bd-8b88-4c5827d6b74d-metrics-cert\") pod \"lws-controller-manager-5647bd797c-fzbg8\" (UID: \"3201c1d8-e6fc-43bd-8b88-4c5827d6b74d\") " pod="openshift-lws-operator/lws-controller-manager-5647bd797c-fzbg8" Sep 14 02:56:12.141471 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:12.141279 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3201c1d8-e6fc-43bd-8b88-4c5827d6b74d-manager-config\") pod \"lws-controller-manager-5647bd797c-fzbg8\" (UID: \"3201c1d8-e6fc-43bd-8b88-4c5827d6b74d\") " pod="openshift-lws-operator/lws-controller-manager-5647bd797c-fzbg8" Sep 14 02:56:12.141471 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:12.141318 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3201c1d8-e6fc-43bd-8b88-4c5827d6b74d-cert\") pod \"lws-controller-manager-5647bd797c-fzbg8\" (UID: \"3201c1d8-e6fc-43bd-8b88-4c5827d6b74d\") " pod="openshift-lws-operator/lws-controller-manager-5647bd797c-fzbg8" Sep 14 02:56:12.142062 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:12.142030 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3201c1d8-e6fc-43bd-8b88-4c5827d6b74d-manager-config\") pod \"lws-controller-manager-5647bd797c-fzbg8\" (UID: \"3201c1d8-e6fc-43bd-8b88-4c5827d6b74d\") " pod="openshift-lws-operator/lws-controller-manager-5647bd797c-fzbg8" Sep 14 02:56:12.143973 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:12.143946 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3201c1d8-e6fc-43bd-8b88-4c5827d6b74d-metrics-cert\") pod \"lws-controller-manager-5647bd797c-fzbg8\" (UID: \"3201c1d8-e6fc-43bd-8b88-4c5827d6b74d\") " pod="openshift-lws-operator/lws-controller-manager-5647bd797c-fzbg8" Sep 14 02:56:12.144076 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:12.143990 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3201c1d8-e6fc-43bd-8b88-4c5827d6b74d-cert\") pod \"lws-controller-manager-5647bd797c-fzbg8\" (UID: \"3201c1d8-e6fc-43bd-8b88-4c5827d6b74d\") " pod="openshift-lws-operator/lws-controller-manager-5647bd797c-fzbg8" Sep 14 02:56:12.164830 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:12.164787 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57m6p\" (UniqueName: \"kubernetes.io/projected/3201c1d8-e6fc-43bd-8b88-4c5827d6b74d-kube-api-access-57m6p\") pod \"lws-controller-manager-5647bd797c-fzbg8\" (UID: \"3201c1d8-e6fc-43bd-8b88-4c5827d6b74d\") " pod="openshift-lws-operator/lws-controller-manager-5647bd797c-fzbg8" Sep 14 02:56:12.264165 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:12.264126 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5647bd797c-fzbg8" Sep 14 02:56:12.558873 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:12.558846 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5647bd797c-fzbg8"] Sep 14 02:56:12.560711 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:56:12.560682 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3201c1d8_e6fc_43bd_8b88_4c5827d6b74d.slice/crio-9d785527281f5456b9d71cada3c567aff01e1f143e269ea1fca1dec8b0efb3c3 WatchSource:0}: Error finding container 9d785527281f5456b9d71cada3c567aff01e1f143e269ea1fca1dec8b0efb3c3: Status 404 returned error can't find the container with id 9d785527281f5456b9d71cada3c567aff01e1f143e269ea1fca1dec8b0efb3c3 Sep 14 02:56:13.177034 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:13.176997 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5647bd797c-fzbg8" event={"ID":"3201c1d8-e6fc-43bd-8b88-4c5827d6b74d","Type":"ContainerStarted","Data":"9d785527281f5456b9d71cada3c567aff01e1f143e269ea1fca1dec8b0efb3c3"} Sep 14 02:56:14.183118 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:14.183084 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5647bd797c-fzbg8" event={"ID":"3201c1d8-e6fc-43bd-8b88-4c5827d6b74d","Type":"ContainerStarted","Data":"704786af3d299f1b24d2784fc30c207445afed448192abe104032559397e25fb"} Sep 14 02:56:14.183591 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:14.183201 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5647bd797c-fzbg8" Sep 14 02:56:14.208760 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:14.208710 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5647bd797c-fzbg8" podStartSLOduration=1.69562131 podStartE2EDuration="3.208693415s" totalImagesPullingTime="1.513072105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:56:14.207126299 +0000 UTC m=+345.936785416" watchObservedRunningTime="2026-09-14 02:56:14.208693415 +0000 UTC m=+345.938352532" Sep 14 02:56:20.306812 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:20.306725 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:21.598139 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.598094 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:21.616319 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.616277 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz"] Sep 14 02:56:21.619726 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.619698 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" Sep 14 02:56:21.622200 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.622174 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 14 02:56:21.622767 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.622748 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 14 02:56:21.622877 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.622749 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 14 02:56:21.623272 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.623258 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-q2bns\"" Sep 14 02:56:21.623340 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.623295 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 14 02:56:21.645784 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.645746 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz"] Sep 14 02:56:21.726751 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.726709 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f4924af-4868-4817-b105-23e45657a58c-tmp\") pod \"opendatahub-operator-controller-manager-6f647699c6-5r6gz\" (UID: \"3f4924af-4868-4817-b105-23e45657a58c\") " pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" Sep 14 02:56:21.726951 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.726760 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp2gb\" (UniqueName: \"kubernetes.io/projected/3f4924af-4868-4817-b105-23e45657a58c-kube-api-access-wp2gb\") pod \"opendatahub-operator-controller-manager-6f647699c6-5r6gz\" (UID: \"3f4924af-4868-4817-b105-23e45657a58c\") " pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" Sep 14 02:56:21.726951 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.726834 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f4924af-4868-4817-b105-23e45657a58c-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f647699c6-5r6gz\" (UID: \"3f4924af-4868-4817-b105-23e45657a58c\") " pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" Sep 14 02:56:21.726951 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.726869 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3f4924af-4868-4817-b105-23e45657a58c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f647699c6-5r6gz\" (UID: \"3f4924af-4868-4817-b105-23e45657a58c\") " pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" Sep 14 02:56:21.726951 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.726905 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3f4924af-4868-4817-b105-23e45657a58c-manifests\") pod \"opendatahub-operator-controller-manager-6f647699c6-5r6gz\" (UID: \"3f4924af-4868-4817-b105-23e45657a58c\") " pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" Sep 14 02:56:21.756470 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.756421 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:21.827746 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.827702 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp2gb\" (UniqueName: \"kubernetes.io/projected/3f4924af-4868-4817-b105-23e45657a58c-kube-api-access-wp2gb\") pod \"opendatahub-operator-controller-manager-6f647699c6-5r6gz\" (UID: \"3f4924af-4868-4817-b105-23e45657a58c\") " pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" Sep 14 02:56:21.827952 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.827792 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f4924af-4868-4817-b105-23e45657a58c-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f647699c6-5r6gz\" (UID: \"3f4924af-4868-4817-b105-23e45657a58c\") " pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" Sep 14 02:56:21.827952 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.827837 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3f4924af-4868-4817-b105-23e45657a58c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f647699c6-5r6gz\" (UID: \"3f4924af-4868-4817-b105-23e45657a58c\") " pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" Sep 14 02:56:21.827952 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.827882 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3f4924af-4868-4817-b105-23e45657a58c-manifests\") pod \"opendatahub-operator-controller-manager-6f647699c6-5r6gz\" (UID: \"3f4924af-4868-4817-b105-23e45657a58c\") " pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" Sep 14 02:56:21.827952 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.827919 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f4924af-4868-4817-b105-23e45657a58c-tmp\") pod \"opendatahub-operator-controller-manager-6f647699c6-5r6gz\" (UID: \"3f4924af-4868-4817-b105-23e45657a58c\") " pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" Sep 14 02:56:21.828379 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.828357 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3f4924af-4868-4817-b105-23e45657a58c-manifests\") pod \"opendatahub-operator-controller-manager-6f647699c6-5r6gz\" (UID: \"3f4924af-4868-4817-b105-23e45657a58c\") " pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" Sep 14 02:56:21.830450 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.830421 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f4924af-4868-4817-b105-23e45657a58c-tmp\") pod \"opendatahub-operator-controller-manager-6f647699c6-5r6gz\" (UID: \"3f4924af-4868-4817-b105-23e45657a58c\") " pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" Sep 14 02:56:21.830602 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.830546 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f4924af-4868-4817-b105-23e45657a58c-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f647699c6-5r6gz\" (UID: \"3f4924af-4868-4817-b105-23e45657a58c\") " pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" Sep 14 02:56:21.830775 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.830753 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3f4924af-4868-4817-b105-23e45657a58c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f647699c6-5r6gz\" (UID: \"3f4924af-4868-4817-b105-23e45657a58c\") " pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" Sep 14 02:56:21.860898 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.860815 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp2gb\" (UniqueName: \"kubernetes.io/projected/3f4924af-4868-4817-b105-23e45657a58c-kube-api-access-wp2gb\") pod \"opendatahub-operator-controller-manager-6f647699c6-5r6gz\" (UID: \"3f4924af-4868-4817-b105-23e45657a58c\") " pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" Sep 14 02:56:21.887102 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.887065 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:21.930649 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.930613 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" Sep 14 02:56:21.983953 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:21.983921 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:22.084668 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:22.084626 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:22.196928 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:22.196891 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:22.214504 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:22.214464 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz"] Sep 14 02:56:22.217529 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:56:22.217498 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4924af_4868_4817_b105_23e45657a58c.slice/crio-bad426ee81233e8a46cda1c8baaaea9e98fd58f7b36f23d2102c58b8adcb7e10 WatchSource:0}: Error finding container bad426ee81233e8a46cda1c8baaaea9e98fd58f7b36f23d2102c58b8adcb7e10: Status 404 returned error can't find the container with id bad426ee81233e8a46cda1c8baaaea9e98fd58f7b36f23d2102c58b8adcb7e10 Sep 14 02:56:23.225336 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:23.225283 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" event={"ID":"3f4924af-4868-4817-b105-23e45657a58c","Type":"ContainerStarted","Data":"bad426ee81233e8a46cda1c8baaaea9e98fd58f7b36f23d2102c58b8adcb7e10"} Sep 14 02:56:25.189009 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:25.188978 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5647bd797c-fzbg8" Sep 14 02:56:25.237712 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:25.237670 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" event={"ID":"3f4924af-4868-4817-b105-23e45657a58c","Type":"ContainerStarted","Data":"a57e4cdc3d63bb4dcabc84aba940324102a8a60deb08a428c7d88b3865f4f1e1"} Sep 14 02:56:27.248931 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:27.248899 2564 generic.go:356] "Generic (PLEG): container finished" podID="3f4924af-4868-4817-b105-23e45657a58c" containerID="a57e4cdc3d63bb4dcabc84aba940324102a8a60deb08a428c7d88b3865f4f1e1" exitCode=0 Sep 14 02:56:27.249385 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:27.248963 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" event={"ID":"3f4924af-4868-4817-b105-23e45657a58c","Type":"ContainerDied","Data":"a57e4cdc3d63bb4dcabc84aba940324102a8a60deb08a428c7d88b3865f4f1e1"} Sep 14 02:56:27.982144 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:27.982106 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:28.255169 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:28.255086 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" event={"ID":"3f4924af-4868-4817-b105-23e45657a58c","Type":"ContainerStarted","Data":"551bc62e89f274f9393178794099a748f6ecbce88bfe669cb8ac219a70d69808"} Sep 14 02:56:28.255554 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:28.255313 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" Sep 14 02:56:28.285113 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:28.285055 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" podStartSLOduration=2.936464522 podStartE2EDuration="7.285038298s" totalImagesPullingTime="3.271299978s" totalInitContainerRuntime="1.077273798s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:56:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:56:28.284327953 +0000 UTC m=+360.013987073" watchObservedRunningTime="2026-09-14 02:56:28.285038298 +0000 UTC m=+360.014697417" Sep 14 02:56:39.262194 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:39.262160 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f647699c6-5r6gz" Sep 14 02:56:40.929966 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:40.929926 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:41.114094 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:41.114048 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:41.446892 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:41.446850 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:41.527498 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:41.527449 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-66f955bb57-bjdhw"] Sep 14 02:56:41.543691 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:41.543650 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66f955bb57-bjdhw" Sep 14 02:56:41.546623 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:41.546586 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 14 02:56:41.546767 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:41.546595 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 14 02:56:41.546767 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:41.546670 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 14 02:56:41.547376 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:41.547360 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 14 02:56:41.547426 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:41.547367 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-685jk\"" Sep 14 02:56:41.553350 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:41.553319 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66f955bb57-bjdhw"] Sep 14 02:56:41.663293 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:41.663260 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:41.712924 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:41.712830 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/800ff9e9-2373-4e6b-aafb-cfe2a0fb7126-tls-certs\") pod \"kube-auth-proxy-66f955bb57-bjdhw\" (UID: \"800ff9e9-2373-4e6b-aafb-cfe2a0fb7126\") " pod="openshift-ingress/kube-auth-proxy-66f955bb57-bjdhw" Sep 14 02:56:41.712924 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:41.712878 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmw68\" (UniqueName: \"kubernetes.io/projected/800ff9e9-2373-4e6b-aafb-cfe2a0fb7126-kube-api-access-kmw68\") pod \"kube-auth-proxy-66f955bb57-bjdhw\" (UID: \"800ff9e9-2373-4e6b-aafb-cfe2a0fb7126\") " pod="openshift-ingress/kube-auth-proxy-66f955bb57-bjdhw" Sep 14 02:56:41.712924 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:41.712899 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/800ff9e9-2373-4e6b-aafb-cfe2a0fb7126-tmp\") pod \"kube-auth-proxy-66f955bb57-bjdhw\" (UID: \"800ff9e9-2373-4e6b-aafb-cfe2a0fb7126\") " pod="openshift-ingress/kube-auth-proxy-66f955bb57-bjdhw" Sep 14 02:56:41.814246 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:41.814209 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/800ff9e9-2373-4e6b-aafb-cfe2a0fb7126-tls-certs\") pod \"kube-auth-proxy-66f955bb57-bjdhw\" (UID: \"800ff9e9-2373-4e6b-aafb-cfe2a0fb7126\") " pod="openshift-ingress/kube-auth-proxy-66f955bb57-bjdhw" Sep 14 02:56:41.814246 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:41.814257 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmw68\" (UniqueName: \"kubernetes.io/projected/800ff9e9-2373-4e6b-aafb-cfe2a0fb7126-kube-api-access-kmw68\") pod \"kube-auth-proxy-66f955bb57-bjdhw\" (UID: \"800ff9e9-2373-4e6b-aafb-cfe2a0fb7126\") " pod="openshift-ingress/kube-auth-proxy-66f955bb57-bjdhw" Sep 14 02:56:41.814464 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:41.814278 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/800ff9e9-2373-4e6b-aafb-cfe2a0fb7126-tmp\") pod \"kube-auth-proxy-66f955bb57-bjdhw\" (UID: \"800ff9e9-2373-4e6b-aafb-cfe2a0fb7126\") " pod="openshift-ingress/kube-auth-proxy-66f955bb57-bjdhw" Sep 14 02:56:41.816681 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:41.816650 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/800ff9e9-2373-4e6b-aafb-cfe2a0fb7126-tmp\") pod \"kube-auth-proxy-66f955bb57-bjdhw\" (UID: \"800ff9e9-2373-4e6b-aafb-cfe2a0fb7126\") " pod="openshift-ingress/kube-auth-proxy-66f955bb57-bjdhw" Sep 14 02:56:41.816895 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:41.816875 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/800ff9e9-2373-4e6b-aafb-cfe2a0fb7126-tls-certs\") pod \"kube-auth-proxy-66f955bb57-bjdhw\" (UID: \"800ff9e9-2373-4e6b-aafb-cfe2a0fb7126\") " pod="openshift-ingress/kube-auth-proxy-66f955bb57-bjdhw" Sep 14 02:56:41.830933 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:41.830900 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmw68\" (UniqueName: \"kubernetes.io/projected/800ff9e9-2373-4e6b-aafb-cfe2a0fb7126-kube-api-access-kmw68\") pod \"kube-auth-proxy-66f955bb57-bjdhw\" (UID: \"800ff9e9-2373-4e6b-aafb-cfe2a0fb7126\") " pod="openshift-ingress/kube-auth-proxy-66f955bb57-bjdhw" Sep 14 02:56:41.854708 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:41.854668 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66f955bb57-bjdhw" Sep 14 02:56:42.133941 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:42.133863 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66f955bb57-bjdhw"] Sep 14 02:56:42.137541 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:56:42.137504 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800ff9e9_2373_4e6b_aafb_cfe2a0fb7126.slice/crio-b10e3f79b70db1065a39ff54e7e7d02e2f4129e612a706698263e86cae80d455 WatchSource:0}: Error finding container b10e3f79b70db1065a39ff54e7e7d02e2f4129e612a706698263e86cae80d455: Status 404 returned error can't find the container with id b10e3f79b70db1065a39ff54e7e7d02e2f4129e612a706698263e86cae80d455 Sep 14 02:56:42.321814 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:42.321775 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66f955bb57-bjdhw" event={"ID":"800ff9e9-2373-4e6b-aafb-cfe2a0fb7126","Type":"ContainerStarted","Data":"b10e3f79b70db1065a39ff54e7e7d02e2f4129e612a706698263e86cae80d455"} Sep 14 02:56:43.238918 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:43.238878 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:43.327123 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:43.327085 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:44.399285 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:44.399238 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:45.338546 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:45.338470 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66f955bb57-bjdhw" event={"ID":"800ff9e9-2373-4e6b-aafb-cfe2a0fb7126","Type":"ContainerStarted","Data":"28ec38cf609229cc897606533873f6398a631f88e36358187b396d7f237cf362"} Sep 14 02:56:45.356074 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:45.356017 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-66f955bb57-bjdhw" podStartSLOduration=1.704665818 podStartE2EDuration="4.355998213s" totalImagesPullingTime="2.651332395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:56:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:56:45.354947838 +0000 UTC m=+377.084606972" watchObservedRunningTime="2026-09-14 02:56:45.355998213 +0000 UTC m=+377.085657331" Sep 14 02:56:48.328015 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:48.327976 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:48.464875 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:48.464842 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:49.019937 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:49.019898 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:49.259686 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:49.259645 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:49.797488 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:49.797443 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:52.287906 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:52.287865 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:52.379276 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:52.379242 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:52.491063 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:52.491031 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:52.638255 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:52.638169 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:52.940621 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:52.940589 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:56:53.135303 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:56:53.135269 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:57:02.507624 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:57:02.507543 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:57:02.710968 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:57:02.710920 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:57:03.066549 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:57:03.066509 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:57:03.331307 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:57:03.331196 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:57:39.632808 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:57:39.632777 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:57:39.696371 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:57:39.696337 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:57:50.813755 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:57:50.813673 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:57:50.914966 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:57:50.914927 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:57:50.988905 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:57:50.988876 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:57:53.138460 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:57:53.138420 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:57:53.243624 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:57:53.243581 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:09.111292 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:09.111256 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:11.557290 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:11.557258 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:11.889091 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:11.888993 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:13.432033 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:13.431994 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:13.564018 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:13.563987 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:13.708413 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:13.708310 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:14.501976 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:14.501937 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:14.687390 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:14.687343 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:16.336086 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:16.336046 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:16.645838 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:16.645796 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:16.902369 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:16.902280 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:17.077855 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:17.077824 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:17.186354 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:17.186272 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:17.448014 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:17.447923 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:17.552766 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:17.552728 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:22.313149 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:22.313116 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:22.507211 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:22.507177 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:25.896918 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:25.896872 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:26.063117 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:26.063088 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:27.587159 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:27.587120 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:27.778922 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:27.778886 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:28.459599 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:28.459566 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:28.555527 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:28.555494 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:35.410145 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:35.410107 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:35.518350 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:35.518311 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:35.626498 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:35.626445 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:39.777790 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:39.777752 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj"] Sep 14 02:58:39.781105 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:39.781086 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj" Sep 14 02:58:39.783872 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:39.783843 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 14 02:58:39.784018 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:39.783944 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 14 02:58:39.784078 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:39.784046 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hd99t\"" Sep 14 02:58:39.803362 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:39.803321 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj"] Sep 14 02:58:39.946898 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:39.946838 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f29e9890-1d4d-4c19-b089-d54d991c6d61-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bthpj\" (UID: \"f29e9890-1d4d-4c19-b089-d54d991c6d61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj" Sep 14 02:58:39.946898 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:39.946907 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw4kx\" (UniqueName: \"kubernetes.io/projected/f29e9890-1d4d-4c19-b089-d54d991c6d61-kube-api-access-fw4kx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bthpj\" (UID: \"f29e9890-1d4d-4c19-b089-d54d991c6d61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj" Sep 14 02:58:40.047682 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:40.047576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f29e9890-1d4d-4c19-b089-d54d991c6d61-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bthpj\" (UID: \"f29e9890-1d4d-4c19-b089-d54d991c6d61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj" Sep 14 02:58:40.047682 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:40.047631 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw4kx\" (UniqueName: \"kubernetes.io/projected/f29e9890-1d4d-4c19-b089-d54d991c6d61-kube-api-access-fw4kx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bthpj\" (UID: \"f29e9890-1d4d-4c19-b089-d54d991c6d61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj" Sep 14 02:58:40.048024 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:40.047999 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f29e9890-1d4d-4c19-b089-d54d991c6d61-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bthpj\" (UID: \"f29e9890-1d4d-4c19-b089-d54d991c6d61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj" Sep 14 02:58:40.059613 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:40.059574 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw4kx\" (UniqueName: \"kubernetes.io/projected/f29e9890-1d4d-4c19-b089-d54d991c6d61-kube-api-access-fw4kx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bthpj\" (UID: \"f29e9890-1d4d-4c19-b089-d54d991c6d61\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj" Sep 14 02:58:40.091387 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:40.091342 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj" Sep 14 02:58:40.405136 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:40.405109 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj"] Sep 14 02:58:40.408206 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:58:40.408164 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29e9890_1d4d_4c19_b089_d54d991c6d61.slice/crio-847733ad99b6cc9c0c627ebd84130d7ec9dc5efdb5632fe54de65b895ace171b WatchSource:0}: Error finding container 847733ad99b6cc9c0c627ebd84130d7ec9dc5efdb5632fe54de65b895ace171b: Status 404 returned error can't find the container with id 847733ad99b6cc9c0c627ebd84130d7ec9dc5efdb5632fe54de65b895ace171b Sep 14 02:58:40.853453 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:40.853359 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj" event={"ID":"f29e9890-1d4d-4c19-b089-d54d991c6d61","Type":"ContainerStarted","Data":"847733ad99b6cc9c0c627ebd84130d7ec9dc5efdb5632fe54de65b895ace171b"} Sep 14 02:58:42.254795 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:42.254729 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:42.376966 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:42.376768 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:44.877596 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:44.877555 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj" event={"ID":"f29e9890-1d4d-4c19-b089-d54d991c6d61","Type":"ContainerStarted","Data":"69cb74a5dd43be18e0ac57e01c16610a6751bbb280a73f170d29a07e33652395"} Sep 14 02:58:44.877990 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:44.877767 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj" Sep 14 02:58:44.913991 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:44.913928 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj" podStartSLOduration=1.648179297 podStartE2EDuration="5.913909518s" totalImagesPullingTime="4.265730221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:58:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:58:44.911097978 +0000 UTC m=+496.640757096" watchObservedRunningTime="2026-09-14 02:58:44.913909518 +0000 UTC m=+496.643568634" Sep 14 02:58:45.915907 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:45.915850 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:46.025954 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:46.025917 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:46.150539 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:46.150508 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:46.249026 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:46.248920 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:49.640350 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:49.640312 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:49.705508 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:49.705458 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:55.158970 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:55.158937 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:55.514645 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:55.514456 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:55.617455 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:55.617420 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:55.710328 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:55.710283 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:58:55.885156 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:58:55.885071 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj" Sep 14 02:59:12.356770 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:12.356735 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:59:12.456550 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:12.456511 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:59:14.580143 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:14.580109 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:59:14.679883 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:14.679851 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:59:14.783321 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:14.783285 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:59:15.377921 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:15.377891 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4nc88"] Sep 14 02:59:15.382379 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:15.382358 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-4nc88" Sep 14 02:59:15.384821 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:15.384798 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gwwx2\"" Sep 14 02:59:15.392569 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:15.391581 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4nc88"] Sep 14 02:59:15.413830 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:15.413801 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:59:15.469696 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:15.469667 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7vf2\" (UniqueName: \"kubernetes.io/projected/726d3004-3e24-4bce-81cc-2add1d71dbd7-kube-api-access-z7vf2\") pod \"authorino-7bcfd8f9cd-4nc88\" (UID: \"726d3004-3e24-4bce-81cc-2add1d71dbd7\") " pod="kuadrant-system/authorino-7bcfd8f9cd-4nc88" Sep 14 02:59:15.499783 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:15.499741 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:59:15.570393 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:15.570363 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7vf2\" (UniqueName: \"kubernetes.io/projected/726d3004-3e24-4bce-81cc-2add1d71dbd7-kube-api-access-z7vf2\") pod \"authorino-7bcfd8f9cd-4nc88\" (UID: \"726d3004-3e24-4bce-81cc-2add1d71dbd7\") " pod="kuadrant-system/authorino-7bcfd8f9cd-4nc88" Sep 14 02:59:15.581360 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:15.581329 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7vf2\" (UniqueName: \"kubernetes.io/projected/726d3004-3e24-4bce-81cc-2add1d71dbd7-kube-api-access-z7vf2\") pod \"authorino-7bcfd8f9cd-4nc88\" (UID: \"726d3004-3e24-4bce-81cc-2add1d71dbd7\") " pod="kuadrant-system/authorino-7bcfd8f9cd-4nc88" Sep 14 02:59:15.694093 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:15.694066 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-4nc88" Sep 14 02:59:15.965611 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:15.965581 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4nc88"] Sep 14 02:59:15.968092 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:59:15.968061 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod726d3004_3e24_4bce_81cc_2add1d71dbd7.slice/crio-f1d55cfeaacd19d69016085fd0b1464672e528ec99e9686e3c507d1e30c3ac66 WatchSource:0}: Error finding container f1d55cfeaacd19d69016085fd0b1464672e528ec99e9686e3c507d1e30c3ac66: Status 404 returned error can't find the container with id f1d55cfeaacd19d69016085fd0b1464672e528ec99e9686e3c507d1e30c3ac66 Sep 14 02:59:16.018158 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:16.018121 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-4nc88" event={"ID":"726d3004-3e24-4bce-81cc-2add1d71dbd7","Type":"ContainerStarted","Data":"f1d55cfeaacd19d69016085fd0b1464672e528ec99e9686e3c507d1e30c3ac66"} Sep 14 02:59:18.030543 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:18.030505 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-4nc88" event={"ID":"726d3004-3e24-4bce-81cc-2add1d71dbd7","Type":"ContainerStarted","Data":"4350179173f629521c54c883e5b68f8ab203033c2500bb5ecc4eeb31c615dcd4"} Sep 14 02:59:18.046559 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:18.046456 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-4nc88" podStartSLOduration=1.22428696 podStartE2EDuration="3.046441086s" totalImagesPullingTime="1.822154126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:59:18.045288754 +0000 UTC m=+529.774947867" watchObservedRunningTime="2026-09-14 02:59:18.046441086 +0000 UTC m=+529.776100203" Sep 14 02:59:18.187283 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:18.187251 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4nc88"] Sep 14 02:59:20.040026 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:20.039943 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-4nc88" podUID="726d3004-3e24-4bce-81cc-2add1d71dbd7" containerName="authorino" containerID="cri-o://4350179173f629521c54c883e5b68f8ab203033c2500bb5ecc4eeb31c615dcd4" gracePeriod=30 Sep 14 02:59:20.279418 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:20.279393 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-4nc88" Sep 14 02:59:20.418104 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:20.418047 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7vf2\" (UniqueName: \"kubernetes.io/projected/726d3004-3e24-4bce-81cc-2add1d71dbd7-kube-api-access-z7vf2\") pod \"726d3004-3e24-4bce-81cc-2add1d71dbd7\" (UID: \"726d3004-3e24-4bce-81cc-2add1d71dbd7\") " Sep 14 02:59:20.420361 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:20.420335 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726d3004-3e24-4bce-81cc-2add1d71dbd7-kube-api-access-z7vf2" (OuterVolumeSpecName: "kube-api-access-z7vf2") pod "726d3004-3e24-4bce-81cc-2add1d71dbd7" (UID: "726d3004-3e24-4bce-81cc-2add1d71dbd7"). InnerVolumeSpecName "kube-api-access-z7vf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:59:20.519066 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:20.519025 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7vf2\" (UniqueName: \"kubernetes.io/projected/726d3004-3e24-4bce-81cc-2add1d71dbd7-kube-api-access-z7vf2\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 02:59:20.800436 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:20.800407 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:59:20.860524 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:20.860464 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:59:21.046066 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:21.045967 2564 generic.go:356] "Generic (PLEG): container finished" podID="726d3004-3e24-4bce-81cc-2add1d71dbd7" containerID="4350179173f629521c54c883e5b68f8ab203033c2500bb5ecc4eeb31c615dcd4" exitCode=0 Sep 14 02:59:21.046066 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:21.046005 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-4nc88" Sep 14 02:59:21.046545 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:21.046058 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-4nc88" event={"ID":"726d3004-3e24-4bce-81cc-2add1d71dbd7","Type":"ContainerDied","Data":"4350179173f629521c54c883e5b68f8ab203033c2500bb5ecc4eeb31c615dcd4"} Sep 14 02:59:21.046545 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:21.046099 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-4nc88" event={"ID":"726d3004-3e24-4bce-81cc-2add1d71dbd7","Type":"ContainerDied","Data":"f1d55cfeaacd19d69016085fd0b1464672e528ec99e9686e3c507d1e30c3ac66"} Sep 14 02:59:21.046545 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:21.046114 2564 scope.go:117] "RemoveContainer" containerID="4350179173f629521c54c883e5b68f8ab203033c2500bb5ecc4eeb31c615dcd4" Sep 14 02:59:21.054689 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:21.054669 2564 scope.go:117] "RemoveContainer" containerID="4350179173f629521c54c883e5b68f8ab203033c2500bb5ecc4eeb31c615dcd4" Sep 14 02:59:21.054995 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:59:21.054968 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4350179173f629521c54c883e5b68f8ab203033c2500bb5ecc4eeb31c615dcd4\": container with ID starting with 4350179173f629521c54c883e5b68f8ab203033c2500bb5ecc4eeb31c615dcd4 not found: ID does not exist" containerID="4350179173f629521c54c883e5b68f8ab203033c2500bb5ecc4eeb31c615dcd4" Sep 14 02:59:21.055056 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:21.055004 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4350179173f629521c54c883e5b68f8ab203033c2500bb5ecc4eeb31c615dcd4"} err="failed to get container status \"4350179173f629521c54c883e5b68f8ab203033c2500bb5ecc4eeb31c615dcd4\": rpc error: code = NotFound desc = could not find container \"4350179173f629521c54c883e5b68f8ab203033c2500bb5ecc4eeb31c615dcd4\": container with ID starting with 4350179173f629521c54c883e5b68f8ab203033c2500bb5ecc4eeb31c615dcd4 not found: ID does not exist" Sep 14 02:59:21.061077 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:21.061052 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4nc88"] Sep 14 02:59:21.064695 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:21.064675 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4nc88"] Sep 14 02:59:22.749114 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:22.749069 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="726d3004-3e24-4bce-81cc-2add1d71dbd7" path="/var/lib/kubelet/pods/726d3004-3e24-4bce-81cc-2add1d71dbd7/volumes" Sep 14 02:59:24.909800 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:24.909766 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:59:30.560264 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:30.560229 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:59:30.657880 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:30.657846 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:59:31.051295 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:31.051264 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:59:43.638781 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:43.638741 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:59:43.738125 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:43.738089 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:59:43.845773 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:43.845734 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-n7lq4"] Sep 14 02:59:43.846086 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:43.846072 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="726d3004-3e24-4bce-81cc-2add1d71dbd7" containerName="authorino" Sep 14 02:59:43.846086 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:43.846086 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="726d3004-3e24-4bce-81cc-2add1d71dbd7" containerName="authorino" Sep 14 02:59:43.846171 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:43.846153 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="726d3004-3e24-4bce-81cc-2add1d71dbd7" containerName="authorino" Sep 14 02:59:43.850617 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:43.850595 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-n7lq4" Sep 14 02:59:43.852935 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:43.852917 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gwwx2\"" Sep 14 02:59:43.855837 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:43.855811 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-n7lq4"] Sep 14 02:59:43.911964 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:43.911933 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-n7lq4"] Sep 14 02:59:43.912213 ip-10-0-129-88 kubenswrapper[2564]: E0914 02:59:43.912196 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dxgjt], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-n7lq4" podUID="3fde5b30-ceea-4cc0-ad87-f4f5c98e5eb7" Sep 14 02:59:43.919444 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:43.919422 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxgjt\" (UniqueName: \"kubernetes.io/projected/3fde5b30-ceea-4cc0-ad87-f4f5c98e5eb7-kube-api-access-dxgjt\") pod \"authorino-557f876f69-n7lq4\" (UID: \"3fde5b30-ceea-4cc0-ad87-f4f5c98e5eb7\") " pod="kuadrant-system/authorino-557f876f69-n7lq4" Sep 14 02:59:44.020292 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:44.020250 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxgjt\" (UniqueName: \"kubernetes.io/projected/3fde5b30-ceea-4cc0-ad87-f4f5c98e5eb7-kube-api-access-dxgjt\") pod \"authorino-557f876f69-n7lq4\" (UID: \"3fde5b30-ceea-4cc0-ad87-f4f5c98e5eb7\") " pod="kuadrant-system/authorino-557f876f69-n7lq4" Sep 14 02:59:44.040861 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:44.040835 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxgjt\" (UniqueName: \"kubernetes.io/projected/3fde5b30-ceea-4cc0-ad87-f4f5c98e5eb7-kube-api-access-dxgjt\") pod \"authorino-557f876f69-n7lq4\" (UID: \"3fde5b30-ceea-4cc0-ad87-f4f5c98e5eb7\") " pod="kuadrant-system/authorino-557f876f69-n7lq4" Sep 14 02:59:44.156937 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:44.156901 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-n7lq4" Sep 14 02:59:44.164017 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:44.163954 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-n7lq4" Sep 14 02:59:44.177182 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:44.177149 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7675f9f946-bndl2"] Sep 14 02:59:44.180302 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:44.180284 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7675f9f946-bndl2" Sep 14 02:59:44.183308 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:44.183285 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 14 02:59:44.193286 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:44.193260 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7675f9f946-bndl2"] Sep 14 02:59:44.221011 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:44.220988 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxgjt\" (UniqueName: \"kubernetes.io/projected/3fde5b30-ceea-4cc0-ad87-f4f5c98e5eb7-kube-api-access-dxgjt\") pod \"3fde5b30-ceea-4cc0-ad87-f4f5c98e5eb7\" (UID: \"3fde5b30-ceea-4cc0-ad87-f4f5c98e5eb7\") " Sep 14 02:59:44.221200 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:44.221180 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5j6c\" (UniqueName: \"kubernetes.io/projected/0289c02d-20ad-44d1-bf76-808985921855-kube-api-access-z5j6c\") pod \"authorino-7675f9f946-bndl2\" (UID: \"0289c02d-20ad-44d1-bf76-808985921855\") " pod="kuadrant-system/authorino-7675f9f946-bndl2" Sep 14 02:59:44.221280 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:44.221265 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0289c02d-20ad-44d1-bf76-808985921855-tls-cert\") pod \"authorino-7675f9f946-bndl2\" (UID: \"0289c02d-20ad-44d1-bf76-808985921855\") " pod="kuadrant-system/authorino-7675f9f946-bndl2" Sep 14 02:59:44.223226 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:44.223198 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fde5b30-ceea-4cc0-ad87-f4f5c98e5eb7-kube-api-access-dxgjt" (OuterVolumeSpecName: "kube-api-access-dxgjt") pod "3fde5b30-ceea-4cc0-ad87-f4f5c98e5eb7" (UID: "3fde5b30-ceea-4cc0-ad87-f4f5c98e5eb7"). InnerVolumeSpecName "kube-api-access-dxgjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 02:59:44.322519 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:44.322461 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5j6c\" (UniqueName: \"kubernetes.io/projected/0289c02d-20ad-44d1-bf76-808985921855-kube-api-access-z5j6c\") pod \"authorino-7675f9f946-bndl2\" (UID: \"0289c02d-20ad-44d1-bf76-808985921855\") " pod="kuadrant-system/authorino-7675f9f946-bndl2" Sep 14 02:59:44.322692 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:44.322551 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0289c02d-20ad-44d1-bf76-808985921855-tls-cert\") pod \"authorino-7675f9f946-bndl2\" (UID: \"0289c02d-20ad-44d1-bf76-808985921855\") " pod="kuadrant-system/authorino-7675f9f946-bndl2" Sep 14 02:59:44.322692 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:44.322616 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxgjt\" (UniqueName: \"kubernetes.io/projected/3fde5b30-ceea-4cc0-ad87-f4f5c98e5eb7-kube-api-access-dxgjt\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 02:59:44.325256 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:44.325231 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0289c02d-20ad-44d1-bf76-808985921855-tls-cert\") pod \"authorino-7675f9f946-bndl2\" (UID: \"0289c02d-20ad-44d1-bf76-808985921855\") " pod="kuadrant-system/authorino-7675f9f946-bndl2" Sep 14 02:59:44.330849 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:44.330816 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5j6c\" (UniqueName: \"kubernetes.io/projected/0289c02d-20ad-44d1-bf76-808985921855-kube-api-access-z5j6c\") pod \"authorino-7675f9f946-bndl2\" (UID: \"0289c02d-20ad-44d1-bf76-808985921855\") " pod="kuadrant-system/authorino-7675f9f946-bndl2" Sep 14 02:59:44.489446 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:44.489366 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7675f9f946-bndl2" Sep 14 02:59:44.761113 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:44.761089 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7675f9f946-bndl2"] Sep 14 02:59:44.763017 ip-10-0-129-88 kubenswrapper[2564]: W0914 02:59:44.762988 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0289c02d_20ad_44d1_bf76_808985921855.slice/crio-daef2e0a42dd6e00d5a24a6aafaf29c54f5f8fa0562211d8ffb57272501b075a WatchSource:0}: Error finding container daef2e0a42dd6e00d5a24a6aafaf29c54f5f8fa0562211d8ffb57272501b075a: Status 404 returned error can't find the container with id daef2e0a42dd6e00d5a24a6aafaf29c54f5f8fa0562211d8ffb57272501b075a Sep 14 02:59:45.162680 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:45.162645 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7675f9f946-bndl2" event={"ID":"0289c02d-20ad-44d1-bf76-808985921855","Type":"ContainerStarted","Data":"daef2e0a42dd6e00d5a24a6aafaf29c54f5f8fa0562211d8ffb57272501b075a"} Sep 14 02:59:45.162680 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:45.162658 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-n7lq4" Sep 14 02:59:45.194134 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:45.194091 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-n7lq4"] Sep 14 02:59:45.199555 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:45.197700 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-n7lq4"] Sep 14 02:59:46.168870 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:46.168834 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7675f9f946-bndl2" event={"ID":"0289c02d-20ad-44d1-bf76-808985921855","Type":"ContainerStarted","Data":"d69168ff8f3d155f72f167c5df762a05ca60e75c1a6210a667d245d5009fcb34"} Sep 14 02:59:46.185408 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:46.185356 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7675f9f946-bndl2" podStartSLOduration=1.687621981 podStartE2EDuration="2.185340445s" totalImagesPullingTime="497.718464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 02:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 02:59:46.183981248 +0000 UTC m=+557.913640377" watchObservedRunningTime="2026-09-14 02:59:46.185340445 +0000 UTC m=+557.914999561" Sep 14 02:59:46.750284 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:46.750244 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fde5b30-ceea-4cc0-ad87-f4f5c98e5eb7" path="/var/lib/kubelet/pods/3fde5b30-ceea-4cc0-ad87-f4f5c98e5eb7/volumes" Sep 14 02:59:50.989142 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:50.989103 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:59:51.096085 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:51.096047 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:59:51.198219 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:51.198184 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:59:51.321050 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:51.320958 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:59:54.473283 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:54.473250 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 02:59:54.575386 ip-10-0-129-88 kubenswrapper[2564]: I0914 02:59:54.575345 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:00:25.923516 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:00:25.923461 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:00:26.238322 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:00:26.238216 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:00:27.494362 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:00:27.494322 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:00:27.691949 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:00:27.691906 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:00:27.826765 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:00:27.826673 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:00:28.528177 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:00:28.528139 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:00:28.659506 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:00:28.659447 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:00:28.731301 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:00:28.731274 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bwv6_4369115a-4032-4f5a-a3e4-c02efe665617/ovn-acl-logging/0.log" Sep 14 03:00:28.733544 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:00:28.733519 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bwv6_4369115a-4032-4f5a-a3e4-c02efe665617/ovn-acl-logging/0.log" Sep 14 03:00:37.704032 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:00:37.703988 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:00:37.893550 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:00:37.893508 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:01:08.083709 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:08.083669 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:01:12.187849 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.187817 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx"] Sep 14 03:01:12.191510 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.191455 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:12.196663 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.196633 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 14 03:01:12.196663 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.196652 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 14 03:01:12.196859 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.196655 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-hcz4h\"" Sep 14 03:01:12.196859 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.196678 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 14 03:01:12.218452 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.218426 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx"] Sep 14 03:01:12.240392 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.240365 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:01:12.280913 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.280876 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4105dd3-1534-437d-8fc8-9b10379b2d89-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w9qx\" (UID: \"f4105dd3-1534-437d-8fc8-9b10379b2d89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:12.280913 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.280918 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2n4h\" (UniqueName: \"kubernetes.io/projected/f4105dd3-1534-437d-8fc8-9b10379b2d89-kube-api-access-k2n4h\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w9qx\" (UID: \"f4105dd3-1534-437d-8fc8-9b10379b2d89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:12.281125 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.280988 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4105dd3-1534-437d-8fc8-9b10379b2d89-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w9qx\" (UID: \"f4105dd3-1534-437d-8fc8-9b10379b2d89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:12.281125 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.281031 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4105dd3-1534-437d-8fc8-9b10379b2d89-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w9qx\" (UID: \"f4105dd3-1534-437d-8fc8-9b10379b2d89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:12.281125 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.281051 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4105dd3-1534-437d-8fc8-9b10379b2d89-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w9qx\" (UID: \"f4105dd3-1534-437d-8fc8-9b10379b2d89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:12.281125 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.281065 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4105dd3-1534-437d-8fc8-9b10379b2d89-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w9qx\" (UID: \"f4105dd3-1534-437d-8fc8-9b10379b2d89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:12.382101 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.382061 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4105dd3-1534-437d-8fc8-9b10379b2d89-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w9qx\" (UID: \"f4105dd3-1534-437d-8fc8-9b10379b2d89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:12.382101 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.382103 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4105dd3-1534-437d-8fc8-9b10379b2d89-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w9qx\" (UID: \"f4105dd3-1534-437d-8fc8-9b10379b2d89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:12.382364 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.382126 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4105dd3-1534-437d-8fc8-9b10379b2d89-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w9qx\" (UID: \"f4105dd3-1534-437d-8fc8-9b10379b2d89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:12.382364 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.382189 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4105dd3-1534-437d-8fc8-9b10379b2d89-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w9qx\" (UID: \"f4105dd3-1534-437d-8fc8-9b10379b2d89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:12.382364 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.382221 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2n4h\" (UniqueName: \"kubernetes.io/projected/f4105dd3-1534-437d-8fc8-9b10379b2d89-kube-api-access-k2n4h\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w9qx\" (UID: \"f4105dd3-1534-437d-8fc8-9b10379b2d89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:12.382364 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.382286 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4105dd3-1534-437d-8fc8-9b10379b2d89-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w9qx\" (UID: \"f4105dd3-1534-437d-8fc8-9b10379b2d89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:12.382780 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.382756 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4105dd3-1534-437d-8fc8-9b10379b2d89-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w9qx\" (UID: \"f4105dd3-1534-437d-8fc8-9b10379b2d89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:12.383125 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.382784 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4105dd3-1534-437d-8fc8-9b10379b2d89-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w9qx\" (UID: \"f4105dd3-1534-437d-8fc8-9b10379b2d89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:12.383125 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.382799 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4105dd3-1534-437d-8fc8-9b10379b2d89-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w9qx\" (UID: \"f4105dd3-1534-437d-8fc8-9b10379b2d89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:12.384885 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.384856 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4105dd3-1534-437d-8fc8-9b10379b2d89-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w9qx\" (UID: \"f4105dd3-1534-437d-8fc8-9b10379b2d89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:12.385005 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.384983 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4105dd3-1534-437d-8fc8-9b10379b2d89-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w9qx\" (UID: \"f4105dd3-1534-437d-8fc8-9b10379b2d89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:12.392829 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.392808 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2n4h\" (UniqueName: \"kubernetes.io/projected/f4105dd3-1534-437d-8fc8-9b10379b2d89-kube-api-access-k2n4h\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5w9qx\" (UID: \"f4105dd3-1534-437d-8fc8-9b10379b2d89\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:12.501737 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.501651 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:12.781079 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.781055 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx"] Sep 14 03:01:12.783085 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:01:12.783053 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4105dd3_1534_437d_8fc8_9b10379b2d89.slice/crio-e23987407fc7a12cdc5e099c21463501a9f0cd4b4f9d2ba231a6b2040b375471 WatchSource:0}: Error finding container e23987407fc7a12cdc5e099c21463501a9f0cd4b4f9d2ba231a6b2040b375471: Status 404 returned error can't find the container with id e23987407fc7a12cdc5e099c21463501a9f0cd4b4f9d2ba231a6b2040b375471 Sep 14 03:01:12.784776 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:12.784757 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 03:01:13.567397 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:13.567353 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" event={"ID":"f4105dd3-1534-437d-8fc8-9b10379b2d89","Type":"ContainerStarted","Data":"e23987407fc7a12cdc5e099c21463501a9f0cd4b4f9d2ba231a6b2040b375471"} Sep 14 03:01:14.576811 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:14.576773 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" event={"ID":"f4105dd3-1534-437d-8fc8-9b10379b2d89","Type":"ContainerStarted","Data":"03c0a0f9954046965aaa031f03e4aafc9d629f52e094ca1eb2e5d30a1d0f4c4a"} Sep 14 03:01:14.598442 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:14.598382 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" podStartSLOduration=1.49240084 podStartE2EDuration="2.598362712s" totalImagesPullingTime="1.105961872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:01:14.596956264 +0000 UTC m=+646.326615395" watchObservedRunningTime="2026-09-14 03:01:14.598362712 +0000 UTC m=+646.328021831" Sep 14 03:01:22.502194 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:22.502149 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:22.502600 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:22.502223 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:22.514805 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:22.514785 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:22.624898 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:22.624868 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5w9qx" Sep 14 03:01:35.065623 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:35.065573 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:01:44.993681 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:44.993643 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:01:56.620866 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:01:56.620816 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:02:12.969327 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:12.969288 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:02:12.992119 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:12.992085 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8"] Sep 14 03:02:12.995606 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:12.995587 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:12.998230 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:12.998207 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 14 03:02:13.006728 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.006705 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8"] Sep 14 03:02:13.022795 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.022770 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4748d49e-2842-4cad-be52-d4655399e85f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4zwd8\" (UID: \"4748d49e-2842-4cad-be52-d4655399e85f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:13.022902 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.022810 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4748d49e-2842-4cad-be52-d4655399e85f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4zwd8\" (UID: \"4748d49e-2842-4cad-be52-d4655399e85f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:13.022902 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.022892 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4748d49e-2842-4cad-be52-d4655399e85f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4zwd8\" (UID: \"4748d49e-2842-4cad-be52-d4655399e85f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:13.022987 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.022917 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4748d49e-2842-4cad-be52-d4655399e85f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4zwd8\" (UID: \"4748d49e-2842-4cad-be52-d4655399e85f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:13.022987 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.022976 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27g6p\" (UniqueName: \"kubernetes.io/projected/4748d49e-2842-4cad-be52-d4655399e85f-kube-api-access-27g6p\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4zwd8\" (UID: \"4748d49e-2842-4cad-be52-d4655399e85f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:13.023061 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.022996 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4748d49e-2842-4cad-be52-d4655399e85f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4zwd8\" (UID: \"4748d49e-2842-4cad-be52-d4655399e85f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:13.075127 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.075084 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:02:13.123535 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.123499 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4748d49e-2842-4cad-be52-d4655399e85f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4zwd8\" (UID: \"4748d49e-2842-4cad-be52-d4655399e85f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:13.123699 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.123539 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4748d49e-2842-4cad-be52-d4655399e85f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4zwd8\" (UID: \"4748d49e-2842-4cad-be52-d4655399e85f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:13.123778 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.123731 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27g6p\" (UniqueName: \"kubernetes.io/projected/4748d49e-2842-4cad-be52-d4655399e85f-kube-api-access-27g6p\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4zwd8\" (UID: \"4748d49e-2842-4cad-be52-d4655399e85f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:13.123778 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.123769 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4748d49e-2842-4cad-be52-d4655399e85f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4zwd8\" (UID: \"4748d49e-2842-4cad-be52-d4655399e85f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:13.123889 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.123811 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4748d49e-2842-4cad-be52-d4655399e85f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4zwd8\" (UID: \"4748d49e-2842-4cad-be52-d4655399e85f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:13.123889 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.123838 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4748d49e-2842-4cad-be52-d4655399e85f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4zwd8\" (UID: \"4748d49e-2842-4cad-be52-d4655399e85f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:13.124001 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.123970 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4748d49e-2842-4cad-be52-d4655399e85f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4zwd8\" (UID: \"4748d49e-2842-4cad-be52-d4655399e85f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:13.124186 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.124164 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4748d49e-2842-4cad-be52-d4655399e85f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4zwd8\" (UID: \"4748d49e-2842-4cad-be52-d4655399e85f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:13.124228 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.124175 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4748d49e-2842-4cad-be52-d4655399e85f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4zwd8\" (UID: \"4748d49e-2842-4cad-be52-d4655399e85f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:13.126230 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.126204 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4748d49e-2842-4cad-be52-d4655399e85f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4zwd8\" (UID: \"4748d49e-2842-4cad-be52-d4655399e85f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:13.126403 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.126387 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4748d49e-2842-4cad-be52-d4655399e85f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4zwd8\" (UID: \"4748d49e-2842-4cad-be52-d4655399e85f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:13.133425 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.133401 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27g6p\" (UniqueName: \"kubernetes.io/projected/4748d49e-2842-4cad-be52-d4655399e85f-kube-api-access-27g6p\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4zwd8\" (UID: \"4748d49e-2842-4cad-be52-d4655399e85f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:13.307540 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.307449 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:13.603081 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.603057 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8"] Sep 14 03:02:13.605092 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:02:13.605066 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4748d49e_2842_4cad_be52_d4655399e85f.slice/crio-9f7819c37cc70f66f0eddb26ad380573c8ffc41236942d39616ba104ac7c3fe1 WatchSource:0}: Error finding container 9f7819c37cc70f66f0eddb26ad380573c8ffc41236942d39616ba104ac7c3fe1: Status 404 returned error can't find the container with id 9f7819c37cc70f66f0eddb26ad380573c8ffc41236942d39616ba104ac7c3fe1 Sep 14 03:02:13.849349 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.849319 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" event={"ID":"4748d49e-2842-4cad-be52-d4655399e85f","Type":"ContainerStarted","Data":"2498ab007282fb041fa45233d166adce00d87f0850da0d83418dd496ae12318e"} Sep 14 03:02:13.849469 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.849379 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" event={"ID":"4748d49e-2842-4cad-be52-d4655399e85f","Type":"ContainerStarted","Data":"9f7819c37cc70f66f0eddb26ad380573c8ffc41236942d39616ba104ac7c3fe1"} Sep 14 03:02:13.867832 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:13.867736 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" podStartSLOduration=1.683242657 podStartE2EDuration="1.867716036s" totalImagesPullingTime="184.473379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:02:13.866726699 +0000 UTC m=+705.596385817" watchObservedRunningTime="2026-09-14 03:02:13.867716036 +0000 UTC m=+705.597375154" Sep 14 03:02:23.307957 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:23.307871 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:23.307957 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:23.307959 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:23.320279 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:23.320255 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:23.914556 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:23.914531 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4zwd8" Sep 14 03:02:52.788021 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:02:52.787969 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:03:18.645287 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:18.645250 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:03:32.653453 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:32.653411 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:03:34.296258 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:34.296212 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65dd4bc4dc-x28vm"] Sep 14 03:03:34.299650 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:34.299625 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65dd4bc4dc-x28vm" Sep 14 03:03:34.318052 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:34.318014 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65dd4bc4dc-x28vm"] Sep 14 03:03:34.353810 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:34.353778 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96p7h\" (UniqueName: \"kubernetes.io/projected/9e5d1c0d-0a82-4c66-8013-2d1648113354-kube-api-access-96p7h\") pod \"authorino-65dd4bc4dc-x28vm\" (UID: \"9e5d1c0d-0a82-4c66-8013-2d1648113354\") " pod="kuadrant-system/authorino-65dd4bc4dc-x28vm" Sep 14 03:03:34.353810 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:34.353815 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9e5d1c0d-0a82-4c66-8013-2d1648113354-tls-cert\") pod \"authorino-65dd4bc4dc-x28vm\" (UID: \"9e5d1c0d-0a82-4c66-8013-2d1648113354\") " pod="kuadrant-system/authorino-65dd4bc4dc-x28vm" Sep 14 03:03:34.454664 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:34.454629 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96p7h\" (UniqueName: \"kubernetes.io/projected/9e5d1c0d-0a82-4c66-8013-2d1648113354-kube-api-access-96p7h\") pod \"authorino-65dd4bc4dc-x28vm\" (UID: \"9e5d1c0d-0a82-4c66-8013-2d1648113354\") " pod="kuadrant-system/authorino-65dd4bc4dc-x28vm" Sep 14 03:03:34.454834 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:34.454673 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9e5d1c0d-0a82-4c66-8013-2d1648113354-tls-cert\") pod \"authorino-65dd4bc4dc-x28vm\" (UID: \"9e5d1c0d-0a82-4c66-8013-2d1648113354\") " pod="kuadrant-system/authorino-65dd4bc4dc-x28vm" Sep 14 03:03:34.457208 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:34.457186 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9e5d1c0d-0a82-4c66-8013-2d1648113354-tls-cert\") pod \"authorino-65dd4bc4dc-x28vm\" (UID: \"9e5d1c0d-0a82-4c66-8013-2d1648113354\") " pod="kuadrant-system/authorino-65dd4bc4dc-x28vm" Sep 14 03:03:34.467268 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:34.467246 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96p7h\" (UniqueName: \"kubernetes.io/projected/9e5d1c0d-0a82-4c66-8013-2d1648113354-kube-api-access-96p7h\") pod \"authorino-65dd4bc4dc-x28vm\" (UID: \"9e5d1c0d-0a82-4c66-8013-2d1648113354\") " pod="kuadrant-system/authorino-65dd4bc4dc-x28vm" Sep 14 03:03:34.609037 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:34.608965 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65dd4bc4dc-x28vm" Sep 14 03:03:34.887970 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:34.887942 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65dd4bc4dc-x28vm"] Sep 14 03:03:34.890670 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:03:34.890634 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e5d1c0d_0a82_4c66_8013_2d1648113354.slice/crio-60eafb92f76705c87abf95dddf8075f347212ed02b71c5256190bcefbc50c562 WatchSource:0}: Error finding container 60eafb92f76705c87abf95dddf8075f347212ed02b71c5256190bcefbc50c562: Status 404 returned error can't find the container with id 60eafb92f76705c87abf95dddf8075f347212ed02b71c5256190bcefbc50c562 Sep 14 03:03:35.239625 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:35.239587 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65dd4bc4dc-x28vm" event={"ID":"9e5d1c0d-0a82-4c66-8013-2d1648113354","Type":"ContainerStarted","Data":"60eafb92f76705c87abf95dddf8075f347212ed02b71c5256190bcefbc50c562"} Sep 14 03:03:37.250449 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:37.250412 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65dd4bc4dc-x28vm" event={"ID":"9e5d1c0d-0a82-4c66-8013-2d1648113354","Type":"ContainerStarted","Data":"b9768d5ceb0a5193640cae81d24bb2155dbb26ca9623c21229907caa3bfbb29e"} Sep 14 03:03:37.293254 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:37.293136 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-65dd4bc4dc-x28vm" podStartSLOduration=1.978884034 podStartE2EDuration="3.293112041s" totalImagesPullingTime="1.314228007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:03:37.267926046 +0000 UTC m=+788.997585166" watchObservedRunningTime="2026-09-14 03:03:37.293112041 +0000 UTC m=+789.022771175" Sep 14 03:03:37.293998 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:37.293970 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7675f9f946-bndl2"] Sep 14 03:03:37.294238 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:37.294210 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7675f9f946-bndl2" podUID="0289c02d-20ad-44d1-bf76-808985921855" containerName="authorino" containerID="cri-o://d69168ff8f3d155f72f167c5df762a05ca60e75c1a6210a667d245d5009fcb34" gracePeriod=30 Sep 14 03:03:37.546491 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:37.546448 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7675f9f946-bndl2" Sep 14 03:03:37.589980 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:37.589946 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0289c02d-20ad-44d1-bf76-808985921855-tls-cert\") pod \"0289c02d-20ad-44d1-bf76-808985921855\" (UID: \"0289c02d-20ad-44d1-bf76-808985921855\") " Sep 14 03:03:37.590163 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:37.589998 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5j6c\" (UniqueName: \"kubernetes.io/projected/0289c02d-20ad-44d1-bf76-808985921855-kube-api-access-z5j6c\") pod \"0289c02d-20ad-44d1-bf76-808985921855\" (UID: \"0289c02d-20ad-44d1-bf76-808985921855\") " Sep 14 03:03:37.592370 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:37.592332 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0289c02d-20ad-44d1-bf76-808985921855-kube-api-access-z5j6c" (OuterVolumeSpecName: "kube-api-access-z5j6c") pod "0289c02d-20ad-44d1-bf76-808985921855" (UID: "0289c02d-20ad-44d1-bf76-808985921855"). InnerVolumeSpecName "kube-api-access-z5j6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:03:37.600878 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:37.600849 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0289c02d-20ad-44d1-bf76-808985921855-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "0289c02d-20ad-44d1-bf76-808985921855" (UID: "0289c02d-20ad-44d1-bf76-808985921855"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 03:03:37.657650 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:37.657613 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:03:37.690948 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:37.690911 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5j6c\" (UniqueName: \"kubernetes.io/projected/0289c02d-20ad-44d1-bf76-808985921855-kube-api-access-z5j6c\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:03:37.690948 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:37.690948 2564 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0289c02d-20ad-44d1-bf76-808985921855-tls-cert\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:03:38.081102 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:38.081050 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:03:38.225665 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:38.225631 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:03:38.257043 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:38.256997 2564 generic.go:356] "Generic (PLEG): container finished" podID="0289c02d-20ad-44d1-bf76-808985921855" containerID="d69168ff8f3d155f72f167c5df762a05ca60e75c1a6210a667d245d5009fcb34" exitCode=0 Sep 14 03:03:38.257490 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:38.257054 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7675f9f946-bndl2" Sep 14 03:03:38.257490 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:38.257056 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7675f9f946-bndl2" event={"ID":"0289c02d-20ad-44d1-bf76-808985921855","Type":"ContainerDied","Data":"d69168ff8f3d155f72f167c5df762a05ca60e75c1a6210a667d245d5009fcb34"} Sep 14 03:03:38.257490 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:38.257103 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7675f9f946-bndl2" event={"ID":"0289c02d-20ad-44d1-bf76-808985921855","Type":"ContainerDied","Data":"daef2e0a42dd6e00d5a24a6aafaf29c54f5f8fa0562211d8ffb57272501b075a"} Sep 14 03:03:38.257490 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:38.257134 2564 scope.go:117] "RemoveContainer" containerID="d69168ff8f3d155f72f167c5df762a05ca60e75c1a6210a667d245d5009fcb34" Sep 14 03:03:38.266288 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:38.266273 2564 scope.go:117] "RemoveContainer" containerID="d69168ff8f3d155f72f167c5df762a05ca60e75c1a6210a667d245d5009fcb34" Sep 14 03:03:38.266622 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:03:38.266601 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d69168ff8f3d155f72f167c5df762a05ca60e75c1a6210a667d245d5009fcb34\": container with ID starting with d69168ff8f3d155f72f167c5df762a05ca60e75c1a6210a667d245d5009fcb34 not found: ID does not exist" containerID="d69168ff8f3d155f72f167c5df762a05ca60e75c1a6210a667d245d5009fcb34" Sep 14 03:03:38.266701 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:38.266634 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d69168ff8f3d155f72f167c5df762a05ca60e75c1a6210a667d245d5009fcb34"} err="failed to get container status \"d69168ff8f3d155f72f167c5df762a05ca60e75c1a6210a667d245d5009fcb34\": rpc error: code = NotFound desc = could not find container \"d69168ff8f3d155f72f167c5df762a05ca60e75c1a6210a667d245d5009fcb34\": container with ID starting with d69168ff8f3d155f72f167c5df762a05ca60e75c1a6210a667d245d5009fcb34 not found: ID does not exist" Sep 14 03:03:38.281374 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:38.281345 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7675f9f946-bndl2"] Sep 14 03:03:38.285864 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:38.285844 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7675f9f946-bndl2"] Sep 14 03:03:38.750039 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:38.750006 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0289c02d-20ad-44d1-bf76-808985921855" path="/var/lib/kubelet/pods/0289c02d-20ad-44d1-bf76-808985921855/volumes" Sep 14 03:03:51.913703 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:51.913626 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7cdc6978dc-qbf8q"] Sep 14 03:03:51.914126 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:51.913970 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0289c02d-20ad-44d1-bf76-808985921855" containerName="authorino" Sep 14 03:03:51.914126 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:51.913981 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0289c02d-20ad-44d1-bf76-808985921855" containerName="authorino" Sep 14 03:03:51.914126 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:51.914051 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0289c02d-20ad-44d1-bf76-808985921855" containerName="authorino" Sep 14 03:03:51.918243 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:51.918223 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cdc6978dc-qbf8q" Sep 14 03:03:51.920423 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:51.920403 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Sep 14 03:03:51.920552 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:51.920512 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-mf5ff\"" Sep 14 03:03:51.925914 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:51.925889 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7cdc6978dc-qbf8q"] Sep 14 03:03:52.019311 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:52.019264 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp97s\" (UniqueName: \"kubernetes.io/projected/111f3c71-002c-40c3-a919-af4af7d729d6-kube-api-access-xp97s\") pod \"ai-gateway-controller-7cdc6978dc-qbf8q\" (UID: \"111f3c71-002c-40c3-a919-af4af7d729d6\") " pod="opendatahub/ai-gateway-controller-7cdc6978dc-qbf8q" Sep 14 03:03:52.120582 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:52.120532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp97s\" (UniqueName: \"kubernetes.io/projected/111f3c71-002c-40c3-a919-af4af7d729d6-kube-api-access-xp97s\") pod \"ai-gateway-controller-7cdc6978dc-qbf8q\" (UID: \"111f3c71-002c-40c3-a919-af4af7d729d6\") " pod="opendatahub/ai-gateway-controller-7cdc6978dc-qbf8q" Sep 14 03:03:52.129219 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:52.129184 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp97s\" (UniqueName: \"kubernetes.io/projected/111f3c71-002c-40c3-a919-af4af7d729d6-kube-api-access-xp97s\") pod \"ai-gateway-controller-7cdc6978dc-qbf8q\" (UID: \"111f3c71-002c-40c3-a919-af4af7d729d6\") " pod="opendatahub/ai-gateway-controller-7cdc6978dc-qbf8q" Sep 14 03:03:52.228998 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:52.228906 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cdc6978dc-qbf8q" Sep 14 03:03:52.505577 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:52.505550 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7cdc6978dc-qbf8q"] Sep 14 03:03:52.507142 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:03:52.507113 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111f3c71_002c_40c3_a919_af4af7d729d6.slice/crio-c03bcdfa9e47a32db68c268017e8f977273f295bdcf036d02211cb5f179d12aa WatchSource:0}: Error finding container c03bcdfa9e47a32db68c268017e8f977273f295bdcf036d02211cb5f179d12aa: Status 404 returned error can't find the container with id c03bcdfa9e47a32db68c268017e8f977273f295bdcf036d02211cb5f179d12aa Sep 14 03:03:53.339409 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:53.339365 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cdc6978dc-qbf8q" event={"ID":"111f3c71-002c-40c3-a919-af4af7d729d6","Type":"ContainerStarted","Data":"c03bcdfa9e47a32db68c268017e8f977273f295bdcf036d02211cb5f179d12aa"} Sep 14 03:03:55.310733 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:55.310693 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:03:55.353698 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:55.353659 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cdc6978dc-qbf8q" event={"ID":"111f3c71-002c-40c3-a919-af4af7d729d6","Type":"ContainerStarted","Data":"0274a56a4253e2522601b29e906b3ed67da3e1920a261f8e3289643f6e8b68a0"} Sep 14 03:03:55.353868 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:55.353780 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7cdc6978dc-qbf8q" Sep 14 03:03:55.370441 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:55.370359 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7cdc6978dc-qbf8q" podStartSLOduration=2.018429966 podStartE2EDuration="4.370339351s" totalImagesPullingTime="2.351909385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:03:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:03:55.368933657 +0000 UTC m=+807.098592773" watchObservedRunningTime="2026-09-14 03:03:55.370339351 +0000 UTC m=+807.099998468" Sep 14 03:03:55.387858 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:03:55.387825 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:06.361585 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:06.361552 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7cdc6978dc-qbf8q" Sep 14 03:04:06.531699 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:06.531663 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-76dfc9b5ff-rgrbx"] Sep 14 03:04:06.537914 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:06.537893 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-76dfc9b5ff-rgrbx" Sep 14 03:04:06.543895 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:06.543871 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-76dfc9b5ff-rgrbx"] Sep 14 03:04:06.651219 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:06.651184 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4ndz\" (UniqueName: \"kubernetes.io/projected/913b62ee-5883-4444-8e49-00e24996683a-kube-api-access-t4ndz\") pod \"ai-gateway-controller-76dfc9b5ff-rgrbx\" (UID: \"913b62ee-5883-4444-8e49-00e24996683a\") " pod="opendatahub/ai-gateway-controller-76dfc9b5ff-rgrbx" Sep 14 03:04:06.752410 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:06.752380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4ndz\" (UniqueName: \"kubernetes.io/projected/913b62ee-5883-4444-8e49-00e24996683a-kube-api-access-t4ndz\") pod \"ai-gateway-controller-76dfc9b5ff-rgrbx\" (UID: \"913b62ee-5883-4444-8e49-00e24996683a\") " pod="opendatahub/ai-gateway-controller-76dfc9b5ff-rgrbx" Sep 14 03:04:06.761528 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:06.761501 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4ndz\" (UniqueName: \"kubernetes.io/projected/913b62ee-5883-4444-8e49-00e24996683a-kube-api-access-t4ndz\") pod \"ai-gateway-controller-76dfc9b5ff-rgrbx\" (UID: \"913b62ee-5883-4444-8e49-00e24996683a\") " pod="opendatahub/ai-gateway-controller-76dfc9b5ff-rgrbx" Sep 14 03:04:06.849951 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:06.849924 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-76dfc9b5ff-rgrbx" Sep 14 03:04:07.121087 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:07.121064 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-76dfc9b5ff-rgrbx"] Sep 14 03:04:07.123029 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:04:07.122995 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913b62ee_5883_4444_8e49_00e24996683a.slice/crio-2806e674b78d8b90555c9852beb553134420892831b088146b33b122b2349559 WatchSource:0}: Error finding container 2806e674b78d8b90555c9852beb553134420892831b088146b33b122b2349559: Status 404 returned error can't find the container with id 2806e674b78d8b90555c9852beb553134420892831b088146b33b122b2349559 Sep 14 03:04:07.410471 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:07.410443 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-76dfc9b5ff-rgrbx" event={"ID":"913b62ee-5883-4444-8e49-00e24996683a","Type":"ContainerStarted","Data":"2806e674b78d8b90555c9852beb553134420892831b088146b33b122b2349559"} Sep 14 03:04:08.417947 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:08.417904 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-76dfc9b5ff-rgrbx" event={"ID":"913b62ee-5883-4444-8e49-00e24996683a","Type":"ContainerStarted","Data":"6af45a785d163d228419c04a84a8a98ddc9d07b7d978bd83e5db5c50af0f0871"} Sep 14 03:04:08.418413 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:08.417966 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-76dfc9b5ff-rgrbx" Sep 14 03:04:08.432898 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:08.432846 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-76dfc9b5ff-rgrbx" podStartSLOduration=1.836139078 podStartE2EDuration="2.432833179s" totalImagesPullingTime="596.694101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:04:08.430943143 +0000 UTC m=+820.160602259" watchObservedRunningTime="2026-09-14 03:04:08.432833179 +0000 UTC m=+820.162492295" Sep 14 03:04:13.492519 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:13.492462 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:19.424280 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:19.424250 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-76dfc9b5ff-rgrbx" Sep 14 03:04:19.466132 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:19.466094 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7cdc6978dc-qbf8q"] Sep 14 03:04:19.466363 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:19.466340 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7cdc6978dc-qbf8q" podUID="111f3c71-002c-40c3-a919-af4af7d729d6" containerName="manager" containerID="cri-o://0274a56a4253e2522601b29e906b3ed67da3e1920a261f8e3289643f6e8b68a0" gracePeriod=30 Sep 14 03:04:19.707056 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:19.707029 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cdc6978dc-qbf8q" Sep 14 03:04:19.762801 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:19.762774 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp97s\" (UniqueName: \"kubernetes.io/projected/111f3c71-002c-40c3-a919-af4af7d729d6-kube-api-access-xp97s\") pod \"111f3c71-002c-40c3-a919-af4af7d729d6\" (UID: \"111f3c71-002c-40c3-a919-af4af7d729d6\") " Sep 14 03:04:19.765051 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:19.765027 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111f3c71-002c-40c3-a919-af4af7d729d6-kube-api-access-xp97s" (OuterVolumeSpecName: "kube-api-access-xp97s") pod "111f3c71-002c-40c3-a919-af4af7d729d6" (UID: "111f3c71-002c-40c3-a919-af4af7d729d6"). InnerVolumeSpecName "kube-api-access-xp97s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:04:19.863814 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:19.863790 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp97s\" (UniqueName: \"kubernetes.io/projected/111f3c71-002c-40c3-a919-af4af7d729d6-kube-api-access-xp97s\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:04:20.483230 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:20.483194 2564 generic.go:356] "Generic (PLEG): container finished" podID="111f3c71-002c-40c3-a919-af4af7d729d6" containerID="0274a56a4253e2522601b29e906b3ed67da3e1920a261f8e3289643f6e8b68a0" exitCode=0 Sep 14 03:04:20.483683 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:20.483234 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cdc6978dc-qbf8q" event={"ID":"111f3c71-002c-40c3-a919-af4af7d729d6","Type":"ContainerDied","Data":"0274a56a4253e2522601b29e906b3ed67da3e1920a261f8e3289643f6e8b68a0"} Sep 14 03:04:20.483683 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:20.483243 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cdc6978dc-qbf8q" Sep 14 03:04:20.483683 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:20.483277 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cdc6978dc-qbf8q" event={"ID":"111f3c71-002c-40c3-a919-af4af7d729d6","Type":"ContainerDied","Data":"c03bcdfa9e47a32db68c268017e8f977273f295bdcf036d02211cb5f179d12aa"} Sep 14 03:04:20.483683 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:20.483298 2564 scope.go:117] "RemoveContainer" containerID="0274a56a4253e2522601b29e906b3ed67da3e1920a261f8e3289643f6e8b68a0" Sep 14 03:04:20.492372 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:20.492351 2564 scope.go:117] "RemoveContainer" containerID="0274a56a4253e2522601b29e906b3ed67da3e1920a261f8e3289643f6e8b68a0" Sep 14 03:04:20.492662 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:04:20.492644 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0274a56a4253e2522601b29e906b3ed67da3e1920a261f8e3289643f6e8b68a0\": container with ID starting with 0274a56a4253e2522601b29e906b3ed67da3e1920a261f8e3289643f6e8b68a0 not found: ID does not exist" containerID="0274a56a4253e2522601b29e906b3ed67da3e1920a261f8e3289643f6e8b68a0" Sep 14 03:04:20.492718 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:20.492671 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0274a56a4253e2522601b29e906b3ed67da3e1920a261f8e3289643f6e8b68a0"} err="failed to get container status \"0274a56a4253e2522601b29e906b3ed67da3e1920a261f8e3289643f6e8b68a0\": rpc error: code = NotFound desc = could not find container \"0274a56a4253e2522601b29e906b3ed67da3e1920a261f8e3289643f6e8b68a0\": container with ID starting with 0274a56a4253e2522601b29e906b3ed67da3e1920a261f8e3289643f6e8b68a0 not found: ID does not exist" Sep 14 03:04:20.507387 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:20.507366 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7cdc6978dc-qbf8q"] Sep 14 03:04:20.517396 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:20.517376 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7cdc6978dc-qbf8q"] Sep 14 03:04:20.642273 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:20.642241 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:20.750325 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:20.750252 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="111f3c71-002c-40c3-a919-af4af7d729d6" path="/var/lib/kubelet/pods/111f3c71-002c-40c3-a919-af4af7d729d6/volumes" Sep 14 03:04:20.978683 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:20.978640 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:21.834271 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:21.834237 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:21.927083 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:21.927050 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:22.928393 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:22.928357 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:23.030787 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:23.030753 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:37.756013 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:37.755961 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:38.078516 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:38.078381 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:38.430779 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:38.430737 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:39.169243 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:39.169208 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:39.299227 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:39.299188 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:39.628418 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:39.628341 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:39.726066 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:39.726029 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:39.819076 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:39.819042 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9005-54e678"] Sep 14 03:04:39.819403 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:39.819392 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="111f3c71-002c-40c3-a919-af4af7d729d6" containerName="manager" Sep 14 03:04:39.819445 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:39.819406 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="111f3c71-002c-40c3-a919-af4af7d729d6" containerName="manager" Sep 14 03:04:39.819509 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:39.819500 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="111f3c71-002c-40c3-a919-af4af7d729d6" containerName="manager" Sep 14 03:04:39.823977 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:39.823961 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9005-54e678" Sep 14 03:04:39.826654 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:39.826632 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4ff8v\"" Sep 14 03:04:39.832079 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:39.832055 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9005-54e678"] Sep 14 03:04:39.939804 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:39.939758 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbbfb\" (UniqueName: \"kubernetes.io/projected/1b82054d-3a7f-48aa-b20b-8ecfa05f89c6-kube-api-access-fbbfb\") pod \"test-curl-9005-54e678\" (UID: \"1b82054d-3a7f-48aa-b20b-8ecfa05f89c6\") " pod="opendatahub/test-curl-9005-54e678" Sep 14 03:04:40.040419 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:40.040378 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbbfb\" (UniqueName: \"kubernetes.io/projected/1b82054d-3a7f-48aa-b20b-8ecfa05f89c6-kube-api-access-fbbfb\") pod \"test-curl-9005-54e678\" (UID: \"1b82054d-3a7f-48aa-b20b-8ecfa05f89c6\") " pod="opendatahub/test-curl-9005-54e678" Sep 14 03:04:40.049540 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:40.049512 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbbfb\" (UniqueName: \"kubernetes.io/projected/1b82054d-3a7f-48aa-b20b-8ecfa05f89c6-kube-api-access-fbbfb\") pod \"test-curl-9005-54e678\" (UID: \"1b82054d-3a7f-48aa-b20b-8ecfa05f89c6\") " pod="opendatahub/test-curl-9005-54e678" Sep 14 03:04:40.134522 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:40.134462 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9005-54e678" Sep 14 03:04:40.405979 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:40.405955 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9005-54e678"] Sep 14 03:04:40.407603 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:04:40.407575 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b82054d_3a7f_48aa_b20b_8ecfa05f89c6.slice/crio-238f633848ba26f478bb6e6c3f5e8339c02ce8213fc5202a553a696a52ca1296 WatchSource:0}: Error finding container 238f633848ba26f478bb6e6c3f5e8339c02ce8213fc5202a553a696a52ca1296: Status 404 returned error can't find the container with id 238f633848ba26f478bb6e6c3f5e8339c02ce8213fc5202a553a696a52ca1296 Sep 14 03:04:40.581074 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:40.580609 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9005-54e678" event={"ID":"1b82054d-3a7f-48aa-b20b-8ecfa05f89c6","Type":"ContainerStarted","Data":"238f633848ba26f478bb6e6c3f5e8339c02ce8213fc5202a553a696a52ca1296"} Sep 14 03:04:41.250139 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:41.250090 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:41.359550 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:41.359515 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:42.601063 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:42.601024 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:42.807759 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:42.807720 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:43.075739 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:43.075701 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:43.268488 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:43.268444 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:43.370866 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:43.370778 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:43.400618 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:43.400591 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx"] Sep 14 03:04:43.404347 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:43.404321 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx" Sep 14 03:04:43.407226 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:43.407206 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e467de-dockercfg-ft29f\"" Sep 14 03:04:43.407794 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:43.407780 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e467de\"" Sep 14 03:04:43.421912 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:43.421887 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx"] Sep 14 03:04:43.476723 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:43.476690 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz9pf\" (UniqueName: \"kubernetes.io/projected/31915719-a15a-449f-814f-84f35762a374-kube-api-access-sz9pf\") pod \"payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx\" (UID: \"31915719-a15a-449f-814f-84f35762a374\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx" Sep 14 03:04:43.476886 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:43.476814 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31915719-a15a-449f-814f-84f35762a374-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx\" (UID: \"31915719-a15a-449f-814f-84f35762a374\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx" Sep 14 03:04:43.578214 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:43.578181 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz9pf\" (UniqueName: \"kubernetes.io/projected/31915719-a15a-449f-814f-84f35762a374-kube-api-access-sz9pf\") pod \"payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx\" (UID: \"31915719-a15a-449f-814f-84f35762a374\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx" Sep 14 03:04:43.578399 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:43.578292 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31915719-a15a-449f-814f-84f35762a374-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx\" (UID: \"31915719-a15a-449f-814f-84f35762a374\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx" Sep 14 03:04:43.578953 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:43.578936 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31915719-a15a-449f-814f-84f35762a374-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx\" (UID: \"31915719-a15a-449f-814f-84f35762a374\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx" Sep 14 03:04:43.596288 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:43.596254 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz9pf\" (UniqueName: \"kubernetes.io/projected/31915719-a15a-449f-814f-84f35762a374-kube-api-access-sz9pf\") pod \"payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx\" (UID: \"31915719-a15a-449f-814f-84f35762a374\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx" Sep 14 03:04:43.603786 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:43.603752 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9005-54e678" event={"ID":"1b82054d-3a7f-48aa-b20b-8ecfa05f89c6","Type":"ContainerStarted","Data":"04f13445649d974e9efbbd658b9ec07760f27add1df84aa5baa7e61fa6186710"} Sep 14 03:04:43.623623 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:43.623427 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9005-54e678" podStartSLOduration=2.291603437 podStartE2EDuration="4.623412368s" totalImagesPullingTime="2.331808931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:04:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:04:43.6220836 +0000 UTC m=+855.351742717" watchObservedRunningTime="2026-09-14 03:04:43.623412368 +0000 UTC m=+855.353071486" Sep 14 03:04:43.714228 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:43.714195 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx" Sep 14 03:04:43.999389 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:43.999364 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx"] Sep 14 03:04:44.000750 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:04:44.000720 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31915719_a15a_449f_814f_84f35762a374.slice/crio-3e4953e62daffabe24e8409b38d428422a70f1fde42bf4850ad785b3976e2b74 WatchSource:0}: Error finding container 3e4953e62daffabe24e8409b38d428422a70f1fde42bf4850ad785b3976e2b74: Status 404 returned error can't find the container with id 3e4953e62daffabe24e8409b38d428422a70f1fde42bf4850ad785b3976e2b74 Sep 14 03:04:44.311605 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:44.311528 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:44.316195 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:44.316171 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9005-54e678"] Sep 14 03:04:44.350176 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:44.350148 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9005-54e678"] Sep 14 03:04:44.609511 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:44.609408 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx" event={"ID":"31915719-a15a-449f-814f-84f35762a374","Type":"ContainerStarted","Data":"3e4953e62daffabe24e8409b38d428422a70f1fde42bf4850ad785b3976e2b74"} Sep 14 03:04:44.609912 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:04:44.609555 2564 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9005-54e678" container="test-curl-9005-54e678" Sep 14 03:04:45.616165 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:45.616076 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx" event={"ID":"31915719-a15a-449f-814f-84f35762a374","Type":"ContainerStarted","Data":"35c83e3fa3fb031bad3fc82bfc6f0b1ee7af805cdca111b35576b9d3ae1f2050"} Sep 14 03:04:45.616601 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:45.616167 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9005-54e678" podUID="1b82054d-3a7f-48aa-b20b-8ecfa05f89c6" containerName="test-curl-9005-54e678" containerID="cri-o://04f13445649d974e9efbbd658b9ec07760f27add1df84aa5baa7e61fa6186710" gracePeriod=2 Sep 14 03:04:45.616601 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:45.616308 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx" Sep 14 03:04:45.635850 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:45.635801 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx" podStartSLOduration=1.3853362200000001 podStartE2EDuration="2.635783527s" totalImagesPullingTime="1.250447307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:04:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:04:45.634101213 +0000 UTC m=+857.363760332" watchObservedRunningTime="2026-09-14 03:04:45.635783527 +0000 UTC m=+857.365442644" Sep 14 03:04:46.207967 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:46.207930 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:46.302688 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:46.302656 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:46.410301 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:46.410256 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:46.866443 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:46.866402 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:46.996346 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:46.996305 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:47.183077 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:47.183034 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:47.554240 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:47.554163 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:47.816148 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:47.816090 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9005-54e678" Sep 14 03:04:47.916627 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:47.916592 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbbfb\" (UniqueName: \"kubernetes.io/projected/1b82054d-3a7f-48aa-b20b-8ecfa05f89c6-kube-api-access-fbbfb\") pod \"1b82054d-3a7f-48aa-b20b-8ecfa05f89c6\" (UID: \"1b82054d-3a7f-48aa-b20b-8ecfa05f89c6\") " Sep 14 03:04:47.919303 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:47.919271 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b82054d-3a7f-48aa-b20b-8ecfa05f89c6-kube-api-access-fbbfb" (OuterVolumeSpecName: "kube-api-access-fbbfb") pod "1b82054d-3a7f-48aa-b20b-8ecfa05f89c6" (UID: "1b82054d-3a7f-48aa-b20b-8ecfa05f89c6"). InnerVolumeSpecName "kube-api-access-fbbfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:04:48.018358 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:48.018310 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbbfb\" (UniqueName: \"kubernetes.io/projected/1b82054d-3a7f-48aa-b20b-8ecfa05f89c6-kube-api-access-fbbfb\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:04:48.220643 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:48.220601 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:48.634893 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:48.634808 2564 generic.go:356] "Generic (PLEG): container finished" podID="1b82054d-3a7f-48aa-b20b-8ecfa05f89c6" containerID="04f13445649d974e9efbbd658b9ec07760f27add1df84aa5baa7e61fa6186710" exitCode=137 Sep 14 03:04:48.634893 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:48.634853 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9005-54e678" Sep 14 03:04:48.635080 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:48.634909 2564 scope.go:117] "RemoveContainer" containerID="04f13445649d974e9efbbd658b9ec07760f27add1df84aa5baa7e61fa6186710" Sep 14 03:04:48.644796 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:48.644769 2564 scope.go:117] "RemoveContainer" containerID="04f13445649d974e9efbbd658b9ec07760f27add1df84aa5baa7e61fa6186710" Sep 14 03:04:48.644796 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:04:48.644781 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9005-54e678\" is forbidden: User \"system:node:ip-10-0-129-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-88.ec2.internal' and this object" podUID="1b82054d-3a7f-48aa-b20b-8ecfa05f89c6" pod="opendatahub/test-curl-9005-54e678" Sep 14 03:04:48.645080 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:04:48.645063 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04f13445649d974e9efbbd658b9ec07760f27add1df84aa5baa7e61fa6186710\": container with ID starting with 04f13445649d974e9efbbd658b9ec07760f27add1df84aa5baa7e61fa6186710 not found: ID does not exist" containerID="04f13445649d974e9efbbd658b9ec07760f27add1df84aa5baa7e61fa6186710" Sep 14 03:04:48.645128 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:48.645089 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f13445649d974e9efbbd658b9ec07760f27add1df84aa5baa7e61fa6186710"} err="failed to get container status \"04f13445649d974e9efbbd658b9ec07760f27add1df84aa5baa7e61fa6186710\": rpc error: code = NotFound desc = could not find container \"04f13445649d974e9efbbd658b9ec07760f27add1df84aa5baa7e61fa6186710\": container with ID starting with 04f13445649d974e9efbbd658b9ec07760f27add1df84aa5baa7e61fa6186710 not found: ID does not exist" Sep 14 03:04:48.747638 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:04:48.747605 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9005-54e678\" is forbidden: User \"system:node:ip-10-0-129-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-88.ec2.internal' and this object" podUID="1b82054d-3a7f-48aa-b20b-8ecfa05f89c6" pod="opendatahub/test-curl-9005-54e678" Sep 14 03:04:48.750213 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:48.750173 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b82054d-3a7f-48aa-b20b-8ecfa05f89c6" path="/var/lib/kubelet/pods/1b82054d-3a7f-48aa-b20b-8ecfa05f89c6/volumes" Sep 14 03:04:49.779211 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:49.779173 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:50.451978 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:50.451944 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:50.539165 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:50.539128 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:52.299306 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:52.299273 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:52.403141 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:52.403103 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:52.555040 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:52.554960 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:53.036223 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.036189 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:53.131903 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.131867 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:53.147920 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.147894 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd"] Sep 14 03:04:53.148264 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.148253 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b82054d-3a7f-48aa-b20b-8ecfa05f89c6" containerName="test-curl-9005-54e678" Sep 14 03:04:53.148312 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.148268 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b82054d-3a7f-48aa-b20b-8ecfa05f89c6" containerName="test-curl-9005-54e678" Sep 14 03:04:53.148343 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.148324 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b82054d-3a7f-48aa-b20b-8ecfa05f89c6" containerName="test-curl-9005-54e678" Sep 14 03:04:53.151199 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.151182 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd" Sep 14 03:04:53.153369 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.153347 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c4b97c19-dockercfg-996sj\"" Sep 14 03:04:53.153739 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.153715 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c4b97c19\"" Sep 14 03:04:53.163862 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.163840 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd"] Sep 14 03:04:53.174488 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.174446 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx"] Sep 14 03:04:53.177923 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.177899 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx" Sep 14 03:04:53.188373 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.188350 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx"] Sep 14 03:04:53.266612 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.266581 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5130e4c-a409-44f0-b896-ad5eaffc162c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd\" (UID: \"c5130e4c-a409-44f0-b896-ad5eaffc162c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd" Sep 14 03:04:53.266772 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.266621 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm64b\" (UniqueName: \"kubernetes.io/projected/27c99f0e-a692-400d-979d-d82aa8fd85dd-kube-api-access-qm64b\") pod \"payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx\" (UID: \"27c99f0e-a692-400d-979d-d82aa8fd85dd\") " pod="openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx" Sep 14 03:04:53.266772 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.266645 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c99f0e-a692-400d-979d-d82aa8fd85dd-plugins-config-volume\") pod \"payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx\" (UID: \"27c99f0e-a692-400d-979d-d82aa8fd85dd\") " pod="openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx" Sep 14 03:04:53.266772 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.266735 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbzg5\" (UniqueName: \"kubernetes.io/projected/c5130e4c-a409-44f0-b896-ad5eaffc162c-kube-api-access-gbzg5\") pod \"payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd\" (UID: \"c5130e4c-a409-44f0-b896-ad5eaffc162c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd" Sep 14 03:04:53.367577 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.367469 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5130e4c-a409-44f0-b896-ad5eaffc162c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd\" (UID: \"c5130e4c-a409-44f0-b896-ad5eaffc162c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd" Sep 14 03:04:53.367577 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.367535 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm64b\" (UniqueName: \"kubernetes.io/projected/27c99f0e-a692-400d-979d-d82aa8fd85dd-kube-api-access-qm64b\") pod \"payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx\" (UID: \"27c99f0e-a692-400d-979d-d82aa8fd85dd\") " pod="openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx" Sep 14 03:04:53.367577 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.367555 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c99f0e-a692-400d-979d-d82aa8fd85dd-plugins-config-volume\") pod \"payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx\" (UID: \"27c99f0e-a692-400d-979d-d82aa8fd85dd\") " pod="openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx" Sep 14 03:04:53.368056 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.367585 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbzg5\" (UniqueName: \"kubernetes.io/projected/c5130e4c-a409-44f0-b896-ad5eaffc162c-kube-api-access-gbzg5\") pod \"payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd\" (UID: \"c5130e4c-a409-44f0-b896-ad5eaffc162c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd" Sep 14 03:04:53.368193 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.368173 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5130e4c-a409-44f0-b896-ad5eaffc162c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd\" (UID: \"c5130e4c-a409-44f0-b896-ad5eaffc162c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd" Sep 14 03:04:53.368293 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.368275 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c99f0e-a692-400d-979d-d82aa8fd85dd-plugins-config-volume\") pod \"payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx\" (UID: \"27c99f0e-a692-400d-979d-d82aa8fd85dd\") " pod="openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx" Sep 14 03:04:53.376986 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.376961 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm64b\" (UniqueName: \"kubernetes.io/projected/27c99f0e-a692-400d-979d-d82aa8fd85dd-kube-api-access-qm64b\") pod \"payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx\" (UID: \"27c99f0e-a692-400d-979d-d82aa8fd85dd\") " pod="openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx" Sep 14 03:04:53.377144 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.377119 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbzg5\" (UniqueName: \"kubernetes.io/projected/c5130e4c-a409-44f0-b896-ad5eaffc162c-kube-api-access-gbzg5\") pod \"payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd\" (UID: \"c5130e4c-a409-44f0-b896-ad5eaffc162c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd" Sep 14 03:04:53.434347 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.434299 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:53.461151 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.461124 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd" Sep 14 03:04:53.491603 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.491576 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx" Sep 14 03:04:53.747182 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.747160 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd"] Sep 14 03:04:53.749798 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:04:53.749773 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5130e4c_a409_44f0_b896_ad5eaffc162c.slice/crio-c5558676d45141ca418672bd5251a4fbc383cd4f2e5316b60ba8a17bd9519480 WatchSource:0}: Error finding container c5558676d45141ca418672bd5251a4fbc383cd4f2e5316b60ba8a17bd9519480: Status 404 returned error can't find the container with id c5558676d45141ca418672bd5251a4fbc383cd4f2e5316b60ba8a17bd9519480 Sep 14 03:04:53.777441 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:53.777416 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx"] Sep 14 03:04:53.785778 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:04:53.785751 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c99f0e_a692_400d_979d_d82aa8fd85dd.slice/crio-a2ab5c343854c1fdf77cf7a67f82fdae3de0a2fbd45b1522fbc981046fe731f0 WatchSource:0}: Error finding container a2ab5c343854c1fdf77cf7a67f82fdae3de0a2fbd45b1522fbc981046fe731f0: Status 404 returned error can't find the container with id a2ab5c343854c1fdf77cf7a67f82fdae3de0a2fbd45b1522fbc981046fe731f0 Sep 14 03:04:54.332695 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:54.332662 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:54.457423 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:54.457381 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:54.554694 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:54.554652 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:54.661635 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:54.661605 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:54.665809 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:54.665781 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx" event={"ID":"27c99f0e-a692-400d-979d-d82aa8fd85dd","Type":"ContainerStarted","Data":"71296e3b525652aa3e8ecaf9a82fd835509479c2648d408320d229823d12ee50"} Sep 14 03:04:54.665954 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:54.665845 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx" Sep 14 03:04:54.665954 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:54.665874 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx" event={"ID":"27c99f0e-a692-400d-979d-d82aa8fd85dd","Type":"ContainerStarted","Data":"a2ab5c343854c1fdf77cf7a67f82fdae3de0a2fbd45b1522fbc981046fe731f0"} Sep 14 03:04:54.667495 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:54.667458 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd" event={"ID":"c5130e4c-a409-44f0-b896-ad5eaffc162c","Type":"ContainerStarted","Data":"e7e5eb9661b3793e5488a6e04608865ea7907e8743058b7964be31cd8e5431dd"} Sep 14 03:04:54.667583 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:54.667522 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd" event={"ID":"c5130e4c-a409-44f0-b896-ad5eaffc162c","Type":"ContainerStarted","Data":"c5558676d45141ca418672bd5251a4fbc383cd4f2e5316b60ba8a17bd9519480"} Sep 14 03:04:54.667583 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:54.667550 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd" Sep 14 03:04:54.686607 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:54.686562 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx" podStartSLOduration=1.686547953 podStartE2EDuration="1.686547953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:04:54.686242492 +0000 UTC m=+866.415901611" watchObservedRunningTime="2026-09-14 03:04:54.686547953 +0000 UTC m=+866.416207085" Sep 14 03:04:55.306643 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:55.306608 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:55.867553 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:55.867512 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:56.154702 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:56.154666 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:56.489832 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:56.489726 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:56.623193 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:56.623154 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx" Sep 14 03:04:56.640158 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:56.640088 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd" podStartSLOduration=3.640070055 podStartE2EDuration="3.640070055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:04:54.709584589 +0000 UTC m=+866.439243705" watchObservedRunningTime="2026-09-14 03:04:56.640070055 +0000 UTC m=+868.369729163" Sep 14 03:04:56.743663 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:56.743578 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:04:59.885810 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:04:59.885767 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:00.167256 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:00.167222 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:00.260619 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:00.260579 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:00.609782 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:00.609689 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:01.906699 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:01.906660 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:02.052417 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:02.052372 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:02.308414 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:02.308322 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:02.783494 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:02.783437 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:02.915722 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:02.915691 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:03.148321 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.148285 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47"] Sep 14 03:05:03.152039 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.152015 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47" Sep 14 03:05:03.157582 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.157563 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-bc1117-dockercfg-dmfn6\"" Sep 14 03:05:03.157841 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.157819 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkkqk\" (UniqueName: \"kubernetes.io/projected/86f58e93-1d50-418d-887e-a7fb31419cc3-kube-api-access-gkkqk\") pod \"payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47\" (UID: \"86f58e93-1d50-418d-887e-a7fb31419cc3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47" Sep 14 03:05:03.157928 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.157911 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f58e93-1d50-418d-887e-a7fb31419cc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47\" (UID: \"86f58e93-1d50-418d-887e-a7fb31419cc3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47" Sep 14 03:05:03.158610 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.158591 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-bc1117\"" Sep 14 03:05:03.192681 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.192652 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47"] Sep 14 03:05:03.258504 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.258442 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkkqk\" (UniqueName: \"kubernetes.io/projected/86f58e93-1d50-418d-887e-a7fb31419cc3-kube-api-access-gkkqk\") pod \"payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47\" (UID: \"86f58e93-1d50-418d-887e-a7fb31419cc3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47" Sep 14 03:05:03.258710 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.258525 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f58e93-1d50-418d-887e-a7fb31419cc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47\" (UID: \"86f58e93-1d50-418d-887e-a7fb31419cc3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47" Sep 14 03:05:03.259244 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.259216 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f58e93-1d50-418d-887e-a7fb31419cc3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47\" (UID: \"86f58e93-1d50-418d-887e-a7fb31419cc3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47" Sep 14 03:05:03.273845 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.273812 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkkqk\" (UniqueName: \"kubernetes.io/projected/86f58e93-1d50-418d-887e-a7fb31419cc3-kube-api-access-gkkqk\") pod \"payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47\" (UID: \"86f58e93-1d50-418d-887e-a7fb31419cc3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47" Sep 14 03:05:03.341622 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.341591 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4"] Sep 14 03:05:03.344859 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.344843 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4" Sep 14 03:05:03.359604 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.359569 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f2b9d3-868c-49d3-bfd9-24fd31e81e9d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4\" (UID: \"38f2b9d3-868c-49d3-bfd9-24fd31e81e9d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4" Sep 14 03:05:03.359758 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.359697 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d6cr\" (UniqueName: \"kubernetes.io/projected/38f2b9d3-868c-49d3-bfd9-24fd31e81e9d-kube-api-access-4d6cr\") pod \"payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4\" (UID: \"38f2b9d3-868c-49d3-bfd9-24fd31e81e9d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4" Sep 14 03:05:03.375244 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.375217 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4"] Sep 14 03:05:03.460870 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.460792 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d6cr\" (UniqueName: \"kubernetes.io/projected/38f2b9d3-868c-49d3-bfd9-24fd31e81e9d-kube-api-access-4d6cr\") pod \"payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4\" (UID: \"38f2b9d3-868c-49d3-bfd9-24fd31e81e9d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4" Sep 14 03:05:03.460870 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.460873 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f2b9d3-868c-49d3-bfd9-24fd31e81e9d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4\" (UID: \"38f2b9d3-868c-49d3-bfd9-24fd31e81e9d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4" Sep 14 03:05:03.461692 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.461667 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f2b9d3-868c-49d3-bfd9-24fd31e81e9d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4\" (UID: \"38f2b9d3-868c-49d3-bfd9-24fd31e81e9d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4" Sep 14 03:05:03.462747 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.462719 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47" Sep 14 03:05:03.479333 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.479309 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d6cr\" (UniqueName: \"kubernetes.io/projected/38f2b9d3-868c-49d3-bfd9-24fd31e81e9d-kube-api-access-4d6cr\") pod \"payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4\" (UID: \"38f2b9d3-868c-49d3-bfd9-24fd31e81e9d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4" Sep 14 03:05:03.663114 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.663082 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4" Sep 14 03:05:03.767849 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.767812 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47"] Sep 14 03:05:03.775621 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:05:03.775595 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f58e93_1d50_418d_887e_a7fb31419cc3.slice/crio-115f30385f9890603bd4f1f94c9b1d62885df161566f2541ab7da9936892c8be WatchSource:0}: Error finding container 115f30385f9890603bd4f1f94c9b1d62885df161566f2541ab7da9936892c8be: Status 404 returned error can't find the container with id 115f30385f9890603bd4f1f94c9b1d62885df161566f2541ab7da9936892c8be Sep 14 03:05:03.954642 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:03.954609 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4"] Sep 14 03:05:03.958060 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:05:03.958033 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f2b9d3_868c_49d3_bfd9_24fd31e81e9d.slice/crio-bf4c1a105e02318b55cce30018c7e8c6c4d89881deaf0d6cabd256f3704c37c2 WatchSource:0}: Error finding container bf4c1a105e02318b55cce30018c7e8c6c4d89881deaf0d6cabd256f3704c37c2: Status 404 returned error can't find the container with id bf4c1a105e02318b55cce30018c7e8c6c4d89881deaf0d6cabd256f3704c37c2 Sep 14 03:05:04.702026 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:04.701986 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:04.722988 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:04.722948 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4" event={"ID":"38f2b9d3-868c-49d3-bfd9-24fd31e81e9d","Type":"ContainerStarted","Data":"abb5ffbd28a930ed6673372a784cd9bb27f932c2bfc13fbda1d758392d719c36"} Sep 14 03:05:04.723183 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:04.723013 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4" event={"ID":"38f2b9d3-868c-49d3-bfd9-24fd31e81e9d","Type":"ContainerStarted","Data":"bf4c1a105e02318b55cce30018c7e8c6c4d89881deaf0d6cabd256f3704c37c2"} Sep 14 03:05:04.723183 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:04.723042 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4" Sep 14 03:05:04.724730 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:04.724699 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47" event={"ID":"86f58e93-1d50-418d-887e-a7fb31419cc3","Type":"ContainerStarted","Data":"8a5d918e42b7e1bced37a06956f2621a3d1e594e4f2f97dce3e9578148ac94fd"} Sep 14 03:05:04.724855 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:04.724744 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47" event={"ID":"86f58e93-1d50-418d-887e-a7fb31419cc3","Type":"ContainerStarted","Data":"115f30385f9890603bd4f1f94c9b1d62885df161566f2541ab7da9936892c8be"} Sep 14 03:05:04.724855 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:04.724783 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47" Sep 14 03:05:04.772176 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:04.772121 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4" podStartSLOduration=1.7721063510000001 podStartE2EDuration="1.772106351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:04.770762595 +0000 UTC m=+876.500421712" watchObservedRunningTime="2026-09-14 03:05:04.772106351 +0000 UTC m=+876.501765468" Sep 14 03:05:05.303913 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:05.303876 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:05.453281 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:05.453237 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:05.674518 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:05.674461 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd" Sep 14 03:05:05.674709 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:05.674632 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx" Sep 14 03:05:05.698921 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:05.698875 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47" podStartSLOduration=2.698859068 podStartE2EDuration="2.698859068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:04.818518565 +0000 UTC m=+876.548177682" watchObservedRunningTime="2026-09-14 03:05:05.698859068 +0000 UTC m=+877.428518186" Sep 14 03:05:13.698393 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:13.698354 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:15.355718 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:15.355678 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:15.734843 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:15.734815 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4" Sep 14 03:05:15.735090 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:15.734889 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47" Sep 14 03:05:16.538667 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:16.538631 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:17.554375 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:17.554333 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:18.430152 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:18.430116 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:19.454204 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:19.454150 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:19.923327 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:19.923271 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:20.157835 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:20.157746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:21.546572 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:21.546533 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:22.336661 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:22.336624 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:23.730111 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:23.730067 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:27.327790 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:27.327757 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:28.775217 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:28.775184 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bwv6_4369115a-4032-4f5a-a3e4-c02efe665617/ovn-acl-logging/0.log" Sep 14 03:05:28.775703 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:28.775295 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bwv6_4369115a-4032-4f5a-a3e4-c02efe665617/ovn-acl-logging/0.log" Sep 14 03:05:31.846806 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:31.846763 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:32.736792 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:32.736755 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:32.769076 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:32.769045 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx"] Sep 14 03:05:32.769284 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:32.769263 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx" podUID="27c99f0e-a692-400d-979d-d82aa8fd85dd" containerName="payload-processing" containerID="cri-o://71296e3b525652aa3e8ecaf9a82fd835509479c2648d408320d229823d12ee50" gracePeriod=30 Sep 14 03:05:32.779688 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:32.779646 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd"] Sep 14 03:05:32.779923 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:32.779884 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd" podUID="c5130e4c-a409-44f0-b896-ad5eaffc162c" containerName="payload-pre-processing" containerID="cri-o://e7e5eb9661b3793e5488a6e04608865ea7907e8743058b7964be31cd8e5431dd" gracePeriod=30 Sep 14 03:05:32.871115 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:32.871085 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:33.062110 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.062084 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx" Sep 14 03:05:33.065046 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.065026 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd" Sep 14 03:05:33.085736 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.085712 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:33.135086 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.135056 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qm64b\" (UniqueName: \"kubernetes.io/projected/27c99f0e-a692-400d-979d-d82aa8fd85dd-kube-api-access-qm64b\") pod \"27c99f0e-a692-400d-979d-d82aa8fd85dd\" (UID: \"27c99f0e-a692-400d-979d-d82aa8fd85dd\") " Sep 14 03:05:33.135319 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.135164 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbzg5\" (UniqueName: \"kubernetes.io/projected/c5130e4c-a409-44f0-b896-ad5eaffc162c-kube-api-access-gbzg5\") pod \"c5130e4c-a409-44f0-b896-ad5eaffc162c\" (UID: \"c5130e4c-a409-44f0-b896-ad5eaffc162c\") " Sep 14 03:05:33.135319 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.135197 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c99f0e-a692-400d-979d-d82aa8fd85dd-plugins-config-volume\") pod \"27c99f0e-a692-400d-979d-d82aa8fd85dd\" (UID: \"27c99f0e-a692-400d-979d-d82aa8fd85dd\") " Sep 14 03:05:33.135319 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.135239 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5130e4c-a409-44f0-b896-ad5eaffc162c-plugins-config-volume\") pod \"c5130e4c-a409-44f0-b896-ad5eaffc162c\" (UID: \"c5130e4c-a409-44f0-b896-ad5eaffc162c\") " Sep 14 03:05:33.135611 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.135577 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c99f0e-a692-400d-979d-d82aa8fd85dd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "27c99f0e-a692-400d-979d-d82aa8fd85dd" (UID: "27c99f0e-a692-400d-979d-d82aa8fd85dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:05:33.135739 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.135644 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5130e4c-a409-44f0-b896-ad5eaffc162c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c5130e4c-a409-44f0-b896-ad5eaffc162c" (UID: "c5130e4c-a409-44f0-b896-ad5eaffc162c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:05:33.137505 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.137462 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5130e4c-a409-44f0-b896-ad5eaffc162c-kube-api-access-gbzg5" (OuterVolumeSpecName: "kube-api-access-gbzg5") pod "c5130e4c-a409-44f0-b896-ad5eaffc162c" (UID: "c5130e4c-a409-44f0-b896-ad5eaffc162c"). InnerVolumeSpecName "kube-api-access-gbzg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:05:33.137505 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.137494 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c99f0e-a692-400d-979d-d82aa8fd85dd-kube-api-access-qm64b" (OuterVolumeSpecName: "kube-api-access-qm64b") pod "27c99f0e-a692-400d-979d-d82aa8fd85dd" (UID: "27c99f0e-a692-400d-979d-d82aa8fd85dd"). InnerVolumeSpecName "kube-api-access-qm64b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:05:33.236861 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.236830 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm64b\" (UniqueName: \"kubernetes.io/projected/27c99f0e-a692-400d-979d-d82aa8fd85dd-kube-api-access-qm64b\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:05:33.236861 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.236858 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbzg5\" (UniqueName: \"kubernetes.io/projected/c5130e4c-a409-44f0-b896-ad5eaffc162c-kube-api-access-gbzg5\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:05:33.236861 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.236868 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27c99f0e-a692-400d-979d-d82aa8fd85dd-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:05:33.237092 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.236876 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5130e4c-a409-44f0-b896-ad5eaffc162c-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:05:33.277733 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.277427 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:33.445634 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.445597 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:33.750715 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.750632 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:33.873928 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.873890 2564 generic.go:356] "Generic (PLEG): container finished" podID="c5130e4c-a409-44f0-b896-ad5eaffc162c" containerID="e7e5eb9661b3793e5488a6e04608865ea7907e8743058b7964be31cd8e5431dd" exitCode=0 Sep 14 03:05:33.874384 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.873944 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd" Sep 14 03:05:33.874384 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.873975 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd" event={"ID":"c5130e4c-a409-44f0-b896-ad5eaffc162c","Type":"ContainerDied","Data":"e7e5eb9661b3793e5488a6e04608865ea7907e8743058b7964be31cd8e5431dd"} Sep 14 03:05:33.874384 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.874033 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd" event={"ID":"c5130e4c-a409-44f0-b896-ad5eaffc162c","Type":"ContainerDied","Data":"c5558676d45141ca418672bd5251a4fbc383cd4f2e5316b60ba8a17bd9519480"} Sep 14 03:05:33.874384 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.874057 2564 scope.go:117] "RemoveContainer" containerID="e7e5eb9661b3793e5488a6e04608865ea7907e8743058b7964be31cd8e5431dd" Sep 14 03:05:33.876378 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.876341 2564 generic.go:356] "Generic (PLEG): container finished" podID="27c99f0e-a692-400d-979d-d82aa8fd85dd" containerID="71296e3b525652aa3e8ecaf9a82fd835509479c2648d408320d229823d12ee50" exitCode=0 Sep 14 03:05:33.876521 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.876406 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx" event={"ID":"27c99f0e-a692-400d-979d-d82aa8fd85dd","Type":"ContainerDied","Data":"71296e3b525652aa3e8ecaf9a82fd835509479c2648d408320d229823d12ee50"} Sep 14 03:05:33.876521 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.876434 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx" Sep 14 03:05:33.876656 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.876446 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx" event={"ID":"27c99f0e-a692-400d-979d-d82aa8fd85dd","Type":"ContainerDied","Data":"a2ab5c343854c1fdf77cf7a67f82fdae3de0a2fbd45b1522fbc981046fe731f0"} Sep 14 03:05:33.888531 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.888052 2564 scope.go:117] "RemoveContainer" containerID="e7e5eb9661b3793e5488a6e04608865ea7907e8743058b7964be31cd8e5431dd" Sep 14 03:05:33.888832 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:05:33.888798 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7e5eb9661b3793e5488a6e04608865ea7907e8743058b7964be31cd8e5431dd\": container with ID starting with e7e5eb9661b3793e5488a6e04608865ea7907e8743058b7964be31cd8e5431dd not found: ID does not exist" containerID="e7e5eb9661b3793e5488a6e04608865ea7907e8743058b7964be31cd8e5431dd" Sep 14 03:05:33.888987 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.888964 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7e5eb9661b3793e5488a6e04608865ea7907e8743058b7964be31cd8e5431dd"} err="failed to get container status \"e7e5eb9661b3793e5488a6e04608865ea7907e8743058b7964be31cd8e5431dd\": rpc error: code = NotFound desc = could not find container \"e7e5eb9661b3793e5488a6e04608865ea7907e8743058b7964be31cd8e5431dd\": container with ID starting with e7e5eb9661b3793e5488a6e04608865ea7907e8743058b7964be31cd8e5431dd not found: ID does not exist" Sep 14 03:05:33.889084 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.889073 2564 scope.go:117] "RemoveContainer" containerID="71296e3b525652aa3e8ecaf9a82fd835509479c2648d408320d229823d12ee50" Sep 14 03:05:33.900276 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.900256 2564 scope.go:117] "RemoveContainer" containerID="71296e3b525652aa3e8ecaf9a82fd835509479c2648d408320d229823d12ee50" Sep 14 03:05:33.900717 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:05:33.900689 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71296e3b525652aa3e8ecaf9a82fd835509479c2648d408320d229823d12ee50\": container with ID starting with 71296e3b525652aa3e8ecaf9a82fd835509479c2648d408320d229823d12ee50 not found: ID does not exist" containerID="71296e3b525652aa3e8ecaf9a82fd835509479c2648d408320d229823d12ee50" Sep 14 03:05:33.900820 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.900727 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71296e3b525652aa3e8ecaf9a82fd835509479c2648d408320d229823d12ee50"} err="failed to get container status \"71296e3b525652aa3e8ecaf9a82fd835509479c2648d408320d229823d12ee50\": rpc error: code = NotFound desc = could not find container \"71296e3b525652aa3e8ecaf9a82fd835509479c2648d408320d229823d12ee50\": container with ID starting with 71296e3b525652aa3e8ecaf9a82fd835509479c2648d408320d229823d12ee50 not found: ID does not exist" Sep 14 03:05:33.928789 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.928753 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:33.953740 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.953709 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd"] Sep 14 03:05:33.980379 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:33.980348 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c4b97c19-77ccddfbc4-92kjd"] Sep 14 03:05:34.029777 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:34.029699 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx"] Sep 14 03:05:34.042955 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:34.042920 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c4b97c19-7fdf86c6d7-h4wdx"] Sep 14 03:05:34.290869 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:34.290776 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:34.724781 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:34.724744 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:34.749564 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:34.749531 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27c99f0e-a692-400d-979d-d82aa8fd85dd" path="/var/lib/kubelet/pods/27c99f0e-a692-400d-979d-d82aa8fd85dd/volumes" Sep 14 03:05:34.749943 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:34.749929 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5130e4c-a409-44f0-b896-ad5eaffc162c" path="/var/lib/kubelet/pods/c5130e4c-a409-44f0-b896-ad5eaffc162c/volumes" Sep 14 03:05:34.906847 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:34.906805 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:35.567757 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:35.567714 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:36.659795 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:36.659760 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:37.139035 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:37.138941 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:37.236703 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:37.236668 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:37.775567 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:37.775441 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:37.878739 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:37.878697 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:38.006126 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.006090 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:38.451134 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.451101 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:38.527456 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.527415 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8"] Sep 14 03:05:38.527892 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.527872 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27c99f0e-a692-400d-979d-d82aa8fd85dd" containerName="payload-processing" Sep 14 03:05:38.528000 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.527894 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="27c99f0e-a692-400d-979d-d82aa8fd85dd" containerName="payload-processing" Sep 14 03:05:38.528000 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.527915 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5130e4c-a409-44f0-b896-ad5eaffc162c" containerName="payload-pre-processing" Sep 14 03:05:38.528000 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.527923 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5130e4c-a409-44f0-b896-ad5eaffc162c" containerName="payload-pre-processing" Sep 14 03:05:38.528422 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.528035 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5130e4c-a409-44f0-b896-ad5eaffc162c" containerName="payload-pre-processing" Sep 14 03:05:38.528422 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.528054 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="27c99f0e-a692-400d-979d-d82aa8fd85dd" containerName="payload-processing" Sep 14 03:05:38.531233 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.531212 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8" Sep 14 03:05:38.537020 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.536995 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-07ec96b6-dockercfg-7945l\"" Sep 14 03:05:38.537163 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.537137 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-07ec96b6\"" Sep 14 03:05:38.545059 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.545035 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8"] Sep 14 03:05:38.549524 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.549500 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:38.606196 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.606161 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d"] Sep 14 03:05:38.609747 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.609730 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d" Sep 14 03:05:38.625061 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.625040 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d"] Sep 14 03:05:38.686957 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.686920 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6ntm\" (UniqueName: \"kubernetes.io/projected/d9f9b7a6-331e-498e-9752-625ff61cabde-kube-api-access-z6ntm\") pod \"payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8\" (UID: \"d9f9b7a6-331e-498e-9752-625ff61cabde\") " pod="openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8" Sep 14 03:05:38.687149 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.687083 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d4276f-0324-4fba-a101-4d186564b1eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d\" (UID: \"16d4276f-0324-4fba-a101-4d186564b1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d" Sep 14 03:05:38.687149 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.687121 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvtjb\" (UniqueName: \"kubernetes.io/projected/16d4276f-0324-4fba-a101-4d186564b1eb-kube-api-access-fvtjb\") pod \"payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d\" (UID: \"16d4276f-0324-4fba-a101-4d186564b1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d" Sep 14 03:05:38.687269 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.687169 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f9b7a6-331e-498e-9752-625ff61cabde-plugins-config-volume\") pod \"payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8\" (UID: \"d9f9b7a6-331e-498e-9752-625ff61cabde\") " pod="openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8" Sep 14 03:05:38.788699 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.788616 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d4276f-0324-4fba-a101-4d186564b1eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d\" (UID: \"16d4276f-0324-4fba-a101-4d186564b1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d" Sep 14 03:05:38.788699 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.788655 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvtjb\" (UniqueName: \"kubernetes.io/projected/16d4276f-0324-4fba-a101-4d186564b1eb-kube-api-access-fvtjb\") pod \"payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d\" (UID: \"16d4276f-0324-4fba-a101-4d186564b1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d" Sep 14 03:05:38.788699 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.788682 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f9b7a6-331e-498e-9752-625ff61cabde-plugins-config-volume\") pod \"payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8\" (UID: \"d9f9b7a6-331e-498e-9752-625ff61cabde\") " pod="openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8" Sep 14 03:05:38.789141 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.788719 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6ntm\" (UniqueName: \"kubernetes.io/projected/d9f9b7a6-331e-498e-9752-625ff61cabde-kube-api-access-z6ntm\") pod \"payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8\" (UID: \"d9f9b7a6-331e-498e-9752-625ff61cabde\") " pod="openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8" Sep 14 03:05:38.789400 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.789380 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d4276f-0324-4fba-a101-4d186564b1eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d\" (UID: \"16d4276f-0324-4fba-a101-4d186564b1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d" Sep 14 03:05:38.789593 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.789570 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f9b7a6-331e-498e-9752-625ff61cabde-plugins-config-volume\") pod \"payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8\" (UID: \"d9f9b7a6-331e-498e-9752-625ff61cabde\") " pod="openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8" Sep 14 03:05:38.803421 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.803400 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvtjb\" (UniqueName: \"kubernetes.io/projected/16d4276f-0324-4fba-a101-4d186564b1eb-kube-api-access-fvtjb\") pod \"payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d\" (UID: \"16d4276f-0324-4fba-a101-4d186564b1eb\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d" Sep 14 03:05:38.803572 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.803426 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6ntm\" (UniqueName: \"kubernetes.io/projected/d9f9b7a6-331e-498e-9752-625ff61cabde-kube-api-access-z6ntm\") pod \"payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8\" (UID: \"d9f9b7a6-331e-498e-9752-625ff61cabde\") " pod="openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8" Sep 14 03:05:38.842256 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.842228 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8" Sep 14 03:05:38.919935 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:38.919907 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d" Sep 14 03:05:39.151626 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:39.151596 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8"] Sep 14 03:05:39.153738 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:05:39.153709 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f9b7a6_331e_498e_9752_625ff61cabde.slice/crio-c97bcff62a3fdd41e0efd58fab99dcc240d54f9ce963664a96c9f4b9b08aab47 WatchSource:0}: Error finding container c97bcff62a3fdd41e0efd58fab99dcc240d54f9ce963664a96c9f4b9b08aab47: Status 404 returned error can't find the container with id c97bcff62a3fdd41e0efd58fab99dcc240d54f9ce963664a96c9f4b9b08aab47 Sep 14 03:05:39.225959 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:39.225892 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:39.226138 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:39.226117 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d"] Sep 14 03:05:39.227954 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:05:39.227930 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d4276f_0324_4fba_a101_4d186564b1eb.slice/crio-d8a41f008f6c6d96b323c63e463a8b21333e9323691af17d91939c2b70634e52 WatchSource:0}: Error finding container d8a41f008f6c6d96b323c63e463a8b21333e9323691af17d91939c2b70634e52: Status 404 returned error can't find the container with id d8a41f008f6c6d96b323c63e463a8b21333e9323691af17d91939c2b70634e52 Sep 14 03:05:39.911453 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:39.911422 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d" event={"ID":"16d4276f-0324-4fba-a101-4d186564b1eb","Type":"ContainerStarted","Data":"1d03e881a5c2d32c984b0d2df4abd539e7e65f29e076f4c6d1e053f31fc1e998"} Sep 14 03:05:39.911947 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:39.911497 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d" event={"ID":"16d4276f-0324-4fba-a101-4d186564b1eb","Type":"ContainerStarted","Data":"d8a41f008f6c6d96b323c63e463a8b21333e9323691af17d91939c2b70634e52"} Sep 14 03:05:39.911947 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:39.911526 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d" Sep 14 03:05:39.913114 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:39.913085 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8" event={"ID":"d9f9b7a6-331e-498e-9752-625ff61cabde","Type":"ContainerStarted","Data":"fef75747e3d06b29f9d7ba16b5cd4b29f28240e11b70e247d0cab52515e9a28f"} Sep 14 03:05:39.913239 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:39.913132 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8" event={"ID":"d9f9b7a6-331e-498e-9752-625ff61cabde","Type":"ContainerStarted","Data":"c97bcff62a3fdd41e0efd58fab99dcc240d54f9ce963664a96c9f4b9b08aab47"} Sep 14 03:05:39.913239 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:39.913172 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8" Sep 14 03:05:39.934278 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:39.934231 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d" podStartSLOduration=1.9342178639999998 podStartE2EDuration="1.934217864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:39.931868751 +0000 UTC m=+911.661527859" watchObservedRunningTime="2026-09-14 03:05:39.934217864 +0000 UTC m=+911.663876975" Sep 14 03:05:39.951169 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:39.951129 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8" podStartSLOduration=1.951114367 podStartE2EDuration="1.951114367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:05:39.950061088 +0000 UTC m=+911.679720207" watchObservedRunningTime="2026-09-14 03:05:39.951114367 +0000 UTC m=+911.680773484" Sep 14 03:05:40.139532 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:40.139499 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:40.921075 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:40.921013 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:05:50.920188 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:50.920156 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d" Sep 14 03:05:50.920812 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:50.920259 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8" Sep 14 03:05:54.432348 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:05:54.432310 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:00.333361 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:00.333318 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:00.987168 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:00.987133 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:01.706926 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:01.706876 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:01.972894 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:01.972808 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-07ec96b6-289mc"] Sep 14 03:06:01.976407 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:01.976383 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-07ec96b6-289mc" Sep 14 03:06:01.981864 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:01.981841 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 03:06:01.982022 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:01.981992 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4hf5p\"" Sep 14 03:06:01.993283 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:01.993249 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-07ec96b6-289mc"] Sep 14 03:06:02.007880 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:02.007843 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9fbd06-97a8-4814-bc0f-fc6545394353-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-07ec96b6-289mc\" (UID: \"9c9fbd06-97a8-4814-bc0f-fc6545394353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-07ec96b6-289mc" Sep 14 03:06:02.109152 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:02.109111 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9fbd06-97a8-4814-bc0f-fc6545394353-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-07ec96b6-289mc\" (UID: \"9c9fbd06-97a8-4814-bc0f-fc6545394353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-07ec96b6-289mc" Sep 14 03:06:02.109756 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:02.109733 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9fbd06-97a8-4814-bc0f-fc6545394353-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-07ec96b6-289mc\" (UID: \"9c9fbd06-97a8-4814-bc0f-fc6545394353\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-07ec96b6-289mc" Sep 14 03:06:02.287256 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:02.287167 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-07ec96b6-289mc" Sep 14 03:06:02.567838 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:02.567810 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-07ec96b6-289mc"] Sep 14 03:06:02.569812 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:06:02.569781 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9fbd06_97a8_4814_bc0f_fc6545394353.slice/crio-6ca45ff405b8788b70e55a562080adef25550db0aa57ea1f3e179872a6e7967c WatchSource:0}: Error finding container 6ca45ff405b8788b70e55a562080adef25550db0aa57ea1f3e179872a6e7967c: Status 404 returned error can't find the container with id 6ca45ff405b8788b70e55a562080adef25550db0aa57ea1f3e179872a6e7967c Sep 14 03:06:02.599358 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:02.599315 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:03.033981 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:03.033940 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-07ec96b6-289mc" event={"ID":"9c9fbd06-97a8-4814-bc0f-fc6545394353","Type":"ContainerStarted","Data":"6ca45ff405b8788b70e55a562080adef25550db0aa57ea1f3e179872a6e7967c"} Sep 14 03:06:03.567464 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:03.567404 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:04.181881 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:04.181844 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:05.051662 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:05.051558 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-07ec96b6-289mc" event={"ID":"9c9fbd06-97a8-4814-bc0f-fc6545394353","Type":"ContainerStarted","Data":"d55cf395ae96c8ea118a6a935b0d5c12740de5d9560ac6b110285f385cc2933b"} Sep 14 03:06:05.074745 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:05.074683 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-07ec96b6-289mc" podStartSLOduration=2.1525691670000002 podStartE2EDuration="4.074664126s" totalImagesPullingTime="1.922094959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:06:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:06:05.072454487 +0000 UTC m=+936.802113595" watchObservedRunningTime="2026-09-14 03:06:05.074664126 +0000 UTC m=+936.804323244" Sep 14 03:06:05.200861 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:05.200825 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:06.058659 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:06.058625 2564 generic.go:356] "Generic (PLEG): container finished" podID="9c9fbd06-97a8-4814-bc0f-fc6545394353" containerID="d55cf395ae96c8ea118a6a935b0d5c12740de5d9560ac6b110285f385cc2933b" exitCode=0 Sep 14 03:06:06.058842 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:06.058694 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-07ec96b6-289mc" event={"ID":"9c9fbd06-97a8-4814-bc0f-fc6545394353","Type":"ContainerDied","Data":"d55cf395ae96c8ea118a6a935b0d5c12740de5d9560ac6b110285f385cc2933b"} Sep 14 03:06:07.203626 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:07.203594 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-07ec96b6-289mc" Sep 14 03:06:07.262203 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:07.262163 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9fbd06-97a8-4814-bc0f-fc6545394353-maas-api-service-ca\") pod \"9c9fbd06-97a8-4814-bc0f-fc6545394353\" (UID: \"9c9fbd06-97a8-4814-bc0f-fc6545394353\") " Sep 14 03:06:07.262567 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:07.262542 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9fbd06-97a8-4814-bc0f-fc6545394353-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9c9fbd06-97a8-4814-bc0f-fc6545394353" (UID: "9c9fbd06-97a8-4814-bc0f-fc6545394353"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:06:07.363646 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:07.363551 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c9fbd06-97a8-4814-bc0f-fc6545394353-maas-api-service-ca\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:06:08.070289 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:08.070246 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-07ec96b6-289mc" event={"ID":"9c9fbd06-97a8-4814-bc0f-fc6545394353","Type":"ContainerDied","Data":"6ca45ff405b8788b70e55a562080adef25550db0aa57ea1f3e179872a6e7967c"} Sep 14 03:06:08.070523 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:08.070313 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ca45ff405b8788b70e55a562080adef25550db0aa57ea1f3e179872a6e7967c" Sep 14 03:06:08.070523 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:08.070270 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-07ec96b6-289mc" Sep 14 03:06:08.087009 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:06:08.086978 2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9fbd06_97a8_4814_bc0f_fc6545394353.slice\": RecentStats: unable to find data in memory cache]" Sep 14 03:06:18.343942 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:18.343905 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:18.951652 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:18.951614 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:18.981445 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:18.981416 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8"] Sep 14 03:06:18.981703 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:18.981673 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8" podUID="d9f9b7a6-331e-498e-9752-625ff61cabde" containerName="payload-processing" containerID="cri-o://fef75747e3d06b29f9d7ba16b5cd4b29f28240e11b70e247d0cab52515e9a28f" gracePeriod=30 Sep 14 03:06:18.995255 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:18.995227 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d"] Sep 14 03:06:18.995539 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:18.995467 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d" podUID="16d4276f-0324-4fba-a101-4d186564b1eb" containerName="payload-pre-processing" containerID="cri-o://1d03e881a5c2d32c984b0d2df4abd539e7e65f29e076f4c6d1e053f31fc1e998" gracePeriod=30 Sep 14 03:06:19.054147 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.054117 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:19.130532 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.130496 2564 generic.go:356] "Generic (PLEG): container finished" podID="d9f9b7a6-331e-498e-9752-625ff61cabde" containerID="fef75747e3d06b29f9d7ba16b5cd4b29f28240e11b70e247d0cab52515e9a28f" exitCode=0 Sep 14 03:06:19.130717 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.130579 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8" event={"ID":"d9f9b7a6-331e-498e-9752-625ff61cabde","Type":"ContainerDied","Data":"fef75747e3d06b29f9d7ba16b5cd4b29f28240e11b70e247d0cab52515e9a28f"} Sep 14 03:06:19.132734 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.132620 2564 generic.go:356] "Generic (PLEG): container finished" podID="16d4276f-0324-4fba-a101-4d186564b1eb" containerID="1d03e881a5c2d32c984b0d2df4abd539e7e65f29e076f4c6d1e053f31fc1e998" exitCode=0 Sep 14 03:06:19.132734 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.132676 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d" event={"ID":"16d4276f-0324-4fba-a101-4d186564b1eb","Type":"ContainerDied","Data":"1d03e881a5c2d32c984b0d2df4abd539e7e65f29e076f4c6d1e053f31fc1e998"} Sep 14 03:06:19.182276 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.182238 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:19.260974 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.260941 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-07ec96b6-289mc"] Sep 14 03:06:19.263924 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.263897 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-07ec96b6-289mc"] Sep 14 03:06:19.264249 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.264225 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:19.270249 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.270232 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8" Sep 14 03:06:19.273322 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.273308 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d" Sep 14 03:06:19.338458 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.338422 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:19.378863 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.378837 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6ntm\" (UniqueName: \"kubernetes.io/projected/d9f9b7a6-331e-498e-9752-625ff61cabde-kube-api-access-z6ntm\") pod \"d9f9b7a6-331e-498e-9752-625ff61cabde\" (UID: \"d9f9b7a6-331e-498e-9752-625ff61cabde\") " Sep 14 03:06:19.379297 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.378899 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f9b7a6-331e-498e-9752-625ff61cabde-plugins-config-volume\") pod \"d9f9b7a6-331e-498e-9752-625ff61cabde\" (UID: \"d9f9b7a6-331e-498e-9752-625ff61cabde\") " Sep 14 03:06:19.379297 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.378942 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvtjb\" (UniqueName: \"kubernetes.io/projected/16d4276f-0324-4fba-a101-4d186564b1eb-kube-api-access-fvtjb\") pod \"16d4276f-0324-4fba-a101-4d186564b1eb\" (UID: \"16d4276f-0324-4fba-a101-4d186564b1eb\") " Sep 14 03:06:19.379297 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.378986 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d4276f-0324-4fba-a101-4d186564b1eb-plugins-config-volume\") pod \"16d4276f-0324-4fba-a101-4d186564b1eb\" (UID: \"16d4276f-0324-4fba-a101-4d186564b1eb\") " Sep 14 03:06:19.379724 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.379694 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d4276f-0324-4fba-a101-4d186564b1eb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "16d4276f-0324-4fba-a101-4d186564b1eb" (UID: "16d4276f-0324-4fba-a101-4d186564b1eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:06:19.379824 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.379802 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f9b7a6-331e-498e-9752-625ff61cabde-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d9f9b7a6-331e-498e-9752-625ff61cabde" (UID: "d9f9b7a6-331e-498e-9752-625ff61cabde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:06:19.382149 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.382122 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f9b7a6-331e-498e-9752-625ff61cabde-kube-api-access-z6ntm" (OuterVolumeSpecName: "kube-api-access-z6ntm") pod "d9f9b7a6-331e-498e-9752-625ff61cabde" (UID: "d9f9b7a6-331e-498e-9752-625ff61cabde"). InnerVolumeSpecName "kube-api-access-z6ntm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:06:19.382261 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.382174 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d4276f-0324-4fba-a101-4d186564b1eb-kube-api-access-fvtjb" (OuterVolumeSpecName: "kube-api-access-fvtjb") pod "16d4276f-0324-4fba-a101-4d186564b1eb" (UID: "16d4276f-0324-4fba-a101-4d186564b1eb"). InnerVolumeSpecName "kube-api-access-fvtjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:06:19.432139 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.432104 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:19.479905 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.479877 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6ntm\" (UniqueName: \"kubernetes.io/projected/d9f9b7a6-331e-498e-9752-625ff61cabde-kube-api-access-z6ntm\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:06:19.479905 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.479901 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9f9b7a6-331e-498e-9752-625ff61cabde-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:06:19.480091 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.479912 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvtjb\" (UniqueName: \"kubernetes.io/projected/16d4276f-0324-4fba-a101-4d186564b1eb-kube-api-access-fvtjb\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:06:19.480091 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.479922 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d4276f-0324-4fba-a101-4d186564b1eb-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:06:19.725564 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.725532 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:19.855703 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:19.855670 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:20.141082 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.141006 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d" Sep 14 03:06:20.141082 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.141044 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d" event={"ID":"16d4276f-0324-4fba-a101-4d186564b1eb","Type":"ContainerDied","Data":"d8a41f008f6c6d96b323c63e463a8b21333e9323691af17d91939c2b70634e52"} Sep 14 03:06:20.141293 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.141114 2564 scope.go:117] "RemoveContainer" containerID="1d03e881a5c2d32c984b0d2df4abd539e7e65f29e076f4c6d1e053f31fc1e998" Sep 14 03:06:20.144883 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.144857 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8" event={"ID":"d9f9b7a6-331e-498e-9752-625ff61cabde","Type":"ContainerDied","Data":"c97bcff62a3fdd41e0efd58fab99dcc240d54f9ce963664a96c9f4b9b08aab47"} Sep 14 03:06:20.145004 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.144953 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8" Sep 14 03:06:20.162150 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.162123 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx"] Sep 14 03:06:20.162432 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.162416 2564 scope.go:117] "RemoveContainer" containerID="fef75747e3d06b29f9d7ba16b5cd4b29f28240e11b70e247d0cab52515e9a28f" Sep 14 03:06:20.162654 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.162635 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c9fbd06-97a8-4814-bc0f-fc6545394353" containerName="revoke-keys" Sep 14 03:06:20.162760 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.162657 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c9fbd06-97a8-4814-bc0f-fc6545394353" containerName="revoke-keys" Sep 14 03:06:20.162760 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.162674 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9f9b7a6-331e-498e-9752-625ff61cabde" containerName="payload-processing" Sep 14 03:06:20.162760 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.162683 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9f9b7a6-331e-498e-9752-625ff61cabde" containerName="payload-processing" Sep 14 03:06:20.162760 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.162700 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16d4276f-0324-4fba-a101-4d186564b1eb" containerName="payload-pre-processing" Sep 14 03:06:20.162760 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.162708 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="16d4276f-0324-4fba-a101-4d186564b1eb" containerName="payload-pre-processing" Sep 14 03:06:20.162971 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.162793 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9f9b7a6-331e-498e-9752-625ff61cabde" containerName="payload-processing" Sep 14 03:06:20.162971 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.162808 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="16d4276f-0324-4fba-a101-4d186564b1eb" containerName="payload-pre-processing" Sep 14 03:06:20.162971 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.162818 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c9fbd06-97a8-4814-bc0f-fc6545394353" containerName="revoke-keys" Sep 14 03:06:20.167451 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.167430 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.169784 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.169761 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 14 03:06:20.169871 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.169761 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-89bb5623-openshift-default-dockercfg-v4gj5\"" Sep 14 03:06:20.179538 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.179518 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx"] Sep 14 03:06:20.187437 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.187408 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d"] Sep 14 03:06:20.194807 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.194786 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-07ec96b6-9568f45db-jls8d"] Sep 14 03:06:20.210290 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.210257 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8"] Sep 14 03:06:20.224246 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.224208 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:20.228840 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.228819 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-07ec96b6-8489d8db9d-tqgf8"] Sep 14 03:06:20.288410 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.288378 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-token\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.288610 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.288420 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-workload-socket\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.288610 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.288537 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-podinfo\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.288610 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.288564 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xddh2\" (UniqueName: \"kubernetes.io/projected/15512f1a-697b-4b4a-8e5d-d803e3cb1953-kube-api-access-xddh2\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.288610 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.288586 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-workload-certs\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.288610 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.288610 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-envoy\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.288784 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.288682 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-data\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.288784 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.288739 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-credential-socket\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.288784 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.288756 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istiod-ca-cert\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.389377 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.389345 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-credential-socket\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.389377 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.389378 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istiod-ca-cert\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.389896 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.389425 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-token\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.389896 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.389457 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-workload-socket\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.389896 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.389535 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-podinfo\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.389896 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.389561 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xddh2\" (UniqueName: \"kubernetes.io/projected/15512f1a-697b-4b4a-8e5d-d803e3cb1953-kube-api-access-xddh2\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.389896 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.389589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-workload-certs\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.389896 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.389614 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-envoy\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.389896 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.389703 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-data\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.390229 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.389927 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-workload-socket\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.390229 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.390093 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-data\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.390229 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.390121 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-workload-certs\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.390367 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.390284 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istiod-ca-cert\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.390367 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.390329 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-credential-socket\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.392131 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.392069 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-envoy\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.392299 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.392277 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-podinfo\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.399697 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.399625 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:20.401992 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.401968 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-token\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.404658 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.404639 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xddh2\" (UniqueName: \"kubernetes.io/projected/15512f1a-697b-4b4a-8e5d-d803e3cb1953-kube-api-access-xddh2\") pod \"e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.480538 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.480511 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:20.493113 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.493079 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:20.751223 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.751181 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16d4276f-0324-4fba-a101-4d186564b1eb" path="/var/lib/kubelet/pods/16d4276f-0324-4fba-a101-4d186564b1eb/volumes" Sep 14 03:06:20.751750 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.751717 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c9fbd06-97a8-4814-bc0f-fc6545394353" path="/var/lib/kubelet/pods/9c9fbd06-97a8-4814-bc0f-fc6545394353/volumes" Sep 14 03:06:20.752102 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.752084 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9f9b7a6-331e-498e-9752-625ff61cabde" path="/var/lib/kubelet/pods/d9f9b7a6-331e-498e-9752-625ff61cabde/volumes" Sep 14 03:06:20.759181 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.759159 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx"] Sep 14 03:06:20.761304 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:06:20.761269 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15512f1a_697b_4b4a_8e5d_d803e3cb1953.slice/crio-b3c80507e8b73629ebbf90d34e375dd5f11a2f19982a1a82e7a067aa3010d853 WatchSource:0}: Error finding container b3c80507e8b73629ebbf90d34e375dd5f11a2f19982a1a82e7a067aa3010d853: Status 404 returned error can't find the container with id b3c80507e8b73629ebbf90d34e375dd5f11a2f19982a1a82e7a067aa3010d853 Sep 14 03:06:20.763133 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:20.763117 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 03:06:21.152581 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:21.152541 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" event={"ID":"15512f1a-697b-4b4a-8e5d-d803e3cb1953","Type":"ContainerStarted","Data":"b3c80507e8b73629ebbf90d34e375dd5f11a2f19982a1a82e7a067aa3010d853"} Sep 14 03:06:21.275075 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:21.275047 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:23.849643 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:23.849598 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:06:23.850071 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:23.849687 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:06:23.850071 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:23.849738 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:06:24.170444 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:24.170412 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" event={"ID":"15512f1a-697b-4b4a-8e5d-d803e3cb1953","Type":"ContainerStarted","Data":"8f26373da36c84960542073d35afce22e9dafbb1a9ba363eafbafcb5f676bdfb"} Sep 14 03:06:24.191846 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:24.191793 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" podStartSLOduration=1.105751324 podStartE2EDuration="4.191777417s" totalImagesPullingTime="3.086026093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:06:24.189903524 +0000 UTC m=+955.919562667" watchObservedRunningTime="2026-09-14 03:06:24.191777417 +0000 UTC m=+955.921436533" Sep 14 03:06:24.481408 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:24.481323 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:24.487242 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:24.487212 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:24.913292 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:24.913265 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:25.176074 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:25.175972 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:25.177145 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:25.177120 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:25.608012 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:25.607894 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:25.704048 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:25.704012 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:26.417745 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:26.417711 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:27.145974 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:27.145944 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:27.246288 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:27.246247 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:27.362981 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:27.362952 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:27.782665 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:27.782636 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:27.884536 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:27.884497 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:27.907947 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:27.907913 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8"] Sep 14 03:06:27.916590 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:27.916562 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8" Sep 14 03:06:27.918958 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:27.918933 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-89bb5623-dockercfg-9b6kh\"" Sep 14 03:06:27.919100 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:27.918943 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-89bb5623\"" Sep 14 03:06:27.922443 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:27.922420 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8"] Sep 14 03:06:27.975706 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:27.975672 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7"] Sep 14 03:06:27.979367 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:27.979349 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7" Sep 14 03:06:27.990192 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:27.990168 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7"] Sep 14 03:06:28.065924 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:28.065845 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f679ff05-40fa-43c9-b6bc-c71ae339d666-plugins-config-volume\") pod \"payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8\" (UID: \"f679ff05-40fa-43c9-b6bc-c71ae339d666\") " pod="openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8" Sep 14 03:06:28.065924 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:28.065882 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m264d\" (UniqueName: \"kubernetes.io/projected/f679ff05-40fa-43c9-b6bc-c71ae339d666-kube-api-access-m264d\") pod \"payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8\" (UID: \"f679ff05-40fa-43c9-b6bc-c71ae339d666\") " pod="openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8" Sep 14 03:06:28.166850 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:28.166810 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm62j\" (UniqueName: \"kubernetes.io/projected/1fcb8b38-9617-4d97-89af-f9ead125f4e1-kube-api-access-xm62j\") pod \"payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7\" (UID: \"1fcb8b38-9617-4d97-89af-f9ead125f4e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7" Sep 14 03:06:28.167042 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:28.166882 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f679ff05-40fa-43c9-b6bc-c71ae339d666-plugins-config-volume\") pod \"payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8\" (UID: \"f679ff05-40fa-43c9-b6bc-c71ae339d666\") " pod="openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8" Sep 14 03:06:28.167042 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:28.166915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m264d\" (UniqueName: \"kubernetes.io/projected/f679ff05-40fa-43c9-b6bc-c71ae339d666-kube-api-access-m264d\") pod \"payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8\" (UID: \"f679ff05-40fa-43c9-b6bc-c71ae339d666\") " pod="openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8" Sep 14 03:06:28.167042 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:28.166938 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fcb8b38-9617-4d97-89af-f9ead125f4e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7\" (UID: \"1fcb8b38-9617-4d97-89af-f9ead125f4e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7" Sep 14 03:06:28.167704 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:28.167684 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f679ff05-40fa-43c9-b6bc-c71ae339d666-plugins-config-volume\") pod \"payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8\" (UID: \"f679ff05-40fa-43c9-b6bc-c71ae339d666\") " pod="openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8" Sep 14 03:06:28.175452 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:28.175429 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m264d\" (UniqueName: \"kubernetes.io/projected/f679ff05-40fa-43c9-b6bc-c71ae339d666-kube-api-access-m264d\") pod \"payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8\" (UID: \"f679ff05-40fa-43c9-b6bc-c71ae339d666\") " pod="openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8" Sep 14 03:06:28.229557 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:28.229526 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8" Sep 14 03:06:28.268339 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:28.268291 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fcb8b38-9617-4d97-89af-f9ead125f4e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7\" (UID: \"1fcb8b38-9617-4d97-89af-f9ead125f4e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7" Sep 14 03:06:28.268546 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:28.268457 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm62j\" (UniqueName: \"kubernetes.io/projected/1fcb8b38-9617-4d97-89af-f9ead125f4e1-kube-api-access-xm62j\") pod \"payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7\" (UID: \"1fcb8b38-9617-4d97-89af-f9ead125f4e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7" Sep 14 03:06:28.269106 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:28.269086 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fcb8b38-9617-4d97-89af-f9ead125f4e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7\" (UID: \"1fcb8b38-9617-4d97-89af-f9ead125f4e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7" Sep 14 03:06:28.277025 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:28.277000 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm62j\" (UniqueName: \"kubernetes.io/projected/1fcb8b38-9617-4d97-89af-f9ead125f4e1-kube-api-access-xm62j\") pod \"payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7\" (UID: \"1fcb8b38-9617-4d97-89af-f9ead125f4e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7" Sep 14 03:06:28.291406 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:28.291068 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7" Sep 14 03:06:28.504242 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:28.504216 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8"] Sep 14 03:06:28.505517 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:06:28.505471 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf679ff05_40fa_43c9_b6bc_c71ae339d666.slice/crio-d18655722aea5b33efee2e7716c4d3a98d1dc910c1d1e3ba885b09a9520a239e WatchSource:0}: Error finding container d18655722aea5b33efee2e7716c4d3a98d1dc910c1d1e3ba885b09a9520a239e: Status 404 returned error can't find the container with id d18655722aea5b33efee2e7716c4d3a98d1dc910c1d1e3ba885b09a9520a239e Sep 14 03:06:28.577686 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:28.577657 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7"] Sep 14 03:06:28.579588 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:06:28.579563 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fcb8b38_9617_4d97_89af_f9ead125f4e1.slice/crio-f6511e46d6b7a4720cb6ba49b5c9c30f4f35ad62376c96e38725b0d3f98cc58c WatchSource:0}: Error finding container f6511e46d6b7a4720cb6ba49b5c9c30f4f35ad62376c96e38725b0d3f98cc58c: Status 404 returned error can't find the container with id f6511e46d6b7a4720cb6ba49b5c9c30f4f35ad62376c96e38725b0d3f98cc58c Sep 14 03:06:29.199255 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:29.199217 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7" event={"ID":"1fcb8b38-9617-4d97-89af-f9ead125f4e1","Type":"ContainerStarted","Data":"545db4b69d4cd4eb9ce9c3bdd5eb9e0e11bed3bacaf1a8ecb644a79df133368a"} Sep 14 03:06:29.199745 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:29.199530 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7" event={"ID":"1fcb8b38-9617-4d97-89af-f9ead125f4e1","Type":"ContainerStarted","Data":"f6511e46d6b7a4720cb6ba49b5c9c30f4f35ad62376c96e38725b0d3f98cc58c"} Sep 14 03:06:29.199745 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:29.199567 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7" Sep 14 03:06:29.200979 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:29.200954 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8" event={"ID":"f679ff05-40fa-43c9-b6bc-c71ae339d666","Type":"ContainerStarted","Data":"a37a0ce1fb9e00477bcd81bcbe67cb6d56da19f9a6677d34845d7cc1a097ae73"} Sep 14 03:06:29.201117 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:29.200993 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8" event={"ID":"f679ff05-40fa-43c9-b6bc-c71ae339d666","Type":"ContainerStarted","Data":"d18655722aea5b33efee2e7716c4d3a98d1dc910c1d1e3ba885b09a9520a239e"} Sep 14 03:06:29.201117 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:29.201070 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8" Sep 14 03:06:29.215288 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:29.215193 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7" podStartSLOduration=2.215173835 podStartE2EDuration="2.215173835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:06:29.214829759 +0000 UTC m=+960.944488877" watchObservedRunningTime="2026-09-14 03:06:29.215173835 +0000 UTC m=+960.944832999" Sep 14 03:06:29.232627 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:29.232582 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8" podStartSLOduration=2.232569069 podStartE2EDuration="2.232569069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:06:29.230390981 +0000 UTC m=+960.960050102" watchObservedRunningTime="2026-09-14 03:06:29.232569069 +0000 UTC m=+960.962228235" Sep 14 03:06:40.209252 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:40.209222 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7" Sep 14 03:06:40.209656 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:40.209305 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8" Sep 14 03:06:41.217745 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:41.217699 2564 patch_prober.go:28] interesting pod/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 14 03:06:41.218176 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:41.217774 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" podUID="15512f1a-697b-4b4a-8e5d-d803e3cb1953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 14 03:06:43.281537 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:43.281500 2564 generic.go:356] "Generic (PLEG): container finished" podID="15512f1a-697b-4b4a-8e5d-d803e3cb1953" containerID="8f26373da36c84960542073d35afce22e9dafbb1a9ba363eafbafcb5f676bdfb" exitCode=0 Sep 14 03:06:43.281537 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:43.281516 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" event={"ID":"15512f1a-697b-4b4a-8e5d-d803e3cb1953","Type":"ContainerDied","Data":"8f26373da36c84960542073d35afce22e9dafbb1a9ba363eafbafcb5f676bdfb"} Sep 14 03:06:43.282082 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:43.282065 2564 scope.go:117] "RemoveContainer" containerID="8f26373da36c84960542073d35afce22e9dafbb1a9ba363eafbafcb5f676bdfb" Sep 14 03:06:43.283576 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:43.283541 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:06:43.283643 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:43.283629 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:06:43.283679 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:43.283668 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:06:44.292148 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:44.292093 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" event={"ID":"15512f1a-697b-4b4a-8e5d-d803e3cb1953","Type":"ContainerStarted","Data":"d33b7b98291b48b17a5d48ec982948c88a06d181d03c9a6b317c0b2303c2f5dc"} Sep 14 03:06:44.481170 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:44.481127 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:44.481632 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:44.481599 2564 patch_prober.go:28] interesting pod/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Sep 14 03:06:44.481749 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:44.481676 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" podUID="15512f1a-697b-4b4a-8e5d-d803e3cb1953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Sep 14 03:06:45.481288 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:45.481251 2564 patch_prober.go:28] interesting pod/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Sep 14 03:06:45.481907 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:45.481870 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" podUID="15512f1a-697b-4b4a-8e5d-d803e3cb1953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Sep 14 03:06:46.482153 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:46.482099 2564 patch_prober.go:28] interesting pod/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Sep 14 03:06:46.482832 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:46.482774 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" podUID="15512f1a-697b-4b4a-8e5d-d803e3cb1953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Sep 14 03:06:47.481665 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:47.481618 2564 patch_prober.go:28] interesting pod/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Sep 14 03:06:47.481944 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:47.481691 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" podUID="15512f1a-697b-4b4a-8e5d-d803e3cb1953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Sep 14 03:06:48.481228 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:48.481190 2564 patch_prober.go:28] interesting pod/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Sep 14 03:06:48.481861 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:48.481828 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" podUID="15512f1a-697b-4b4a-8e5d-d803e3cb1953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Sep 14 03:06:49.334752 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:49.334710 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:49.675195 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:49.675113 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:49.676026 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:49.675520 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:49.676337 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:49.676318 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:06:50.629049 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:50.629008 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:50.814277 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:50.814239 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:51.718293 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:51.718253 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:52.483619 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:52.483572 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:53.153539 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:53.153469 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:53.734011 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:53.733966 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:56.946958 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:56.946925 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:57.524987 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:57.524949 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:57.549186 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:57.549153 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8"] Sep 14 03:06:57.549426 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:57.549396 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8" podUID="f679ff05-40fa-43c9-b6bc-c71ae339d666" containerName="payload-processing" containerID="cri-o://a37a0ce1fb9e00477bcd81bcbe67cb6d56da19f9a6677d34845d7cc1a097ae73" gracePeriod=30 Sep 14 03:06:57.559884 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:57.559855 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7"] Sep 14 03:06:57.560072 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:57.560052 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7" podUID="1fcb8b38-9617-4d97-89af-f9ead125f4e1" containerName="payload-pre-processing" containerID="cri-o://545db4b69d4cd4eb9ce9c3bdd5eb9e0e11bed3bacaf1a8ecb644a79df133368a" gracePeriod=30 Sep 14 03:06:57.628069 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:57.628036 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:57.720113 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:57.720079 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:57.796946 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:57.796911 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:57.822710 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:57.822681 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8" Sep 14 03:06:57.846622 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:57.846584 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7" Sep 14 03:06:57.952762 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:57.952727 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fcb8b38-9617-4d97-89af-f9ead125f4e1-plugins-config-volume\") pod \"1fcb8b38-9617-4d97-89af-f9ead125f4e1\" (UID: \"1fcb8b38-9617-4d97-89af-f9ead125f4e1\") " Sep 14 03:06:57.953253 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:57.952826 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f679ff05-40fa-43c9-b6bc-c71ae339d666-plugins-config-volume\") pod \"f679ff05-40fa-43c9-b6bc-c71ae339d666\" (UID: \"f679ff05-40fa-43c9-b6bc-c71ae339d666\") " Sep 14 03:06:57.953253 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:57.952845 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m264d\" (UniqueName: \"kubernetes.io/projected/f679ff05-40fa-43c9-b6bc-c71ae339d666-kube-api-access-m264d\") pod \"f679ff05-40fa-43c9-b6bc-c71ae339d666\" (UID: \"f679ff05-40fa-43c9-b6bc-c71ae339d666\") " Sep 14 03:06:57.953253 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:57.952875 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xm62j\" (UniqueName: \"kubernetes.io/projected/1fcb8b38-9617-4d97-89af-f9ead125f4e1-kube-api-access-xm62j\") pod \"1fcb8b38-9617-4d97-89af-f9ead125f4e1\" (UID: \"1fcb8b38-9617-4d97-89af-f9ead125f4e1\") " Sep 14 03:06:57.953253 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:57.953214 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fcb8b38-9617-4d97-89af-f9ead125f4e1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1fcb8b38-9617-4d97-89af-f9ead125f4e1" (UID: "1fcb8b38-9617-4d97-89af-f9ead125f4e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:06:57.953409 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:57.953271 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f679ff05-40fa-43c9-b6bc-c71ae339d666-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f679ff05-40fa-43c9-b6bc-c71ae339d666" (UID: "f679ff05-40fa-43c9-b6bc-c71ae339d666"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:06:57.955251 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:57.955228 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fcb8b38-9617-4d97-89af-f9ead125f4e1-kube-api-access-xm62j" (OuterVolumeSpecName: "kube-api-access-xm62j") pod "1fcb8b38-9617-4d97-89af-f9ead125f4e1" (UID: "1fcb8b38-9617-4d97-89af-f9ead125f4e1"). InnerVolumeSpecName "kube-api-access-xm62j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:06:57.955324 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:57.955227 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f679ff05-40fa-43c9-b6bc-c71ae339d666-kube-api-access-m264d" (OuterVolumeSpecName: "kube-api-access-m264d") pod "f679ff05-40fa-43c9-b6bc-c71ae339d666" (UID: "f679ff05-40fa-43c9-b6bc-c71ae339d666"). InnerVolumeSpecName "kube-api-access-m264d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:06:58.053639 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.053547 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fcb8b38-9617-4d97-89af-f9ead125f4e1-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:06:58.053639 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.053579 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f679ff05-40fa-43c9-b6bc-c71ae339d666-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:06:58.053639 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.053588 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m264d\" (UniqueName: \"kubernetes.io/projected/f679ff05-40fa-43c9-b6bc-c71ae339d666-kube-api-access-m264d\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:06:58.053639 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.053596 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm62j\" (UniqueName: \"kubernetes.io/projected/1fcb8b38-9617-4d97-89af-f9ead125f4e1-kube-api-access-xm62j\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:06:58.081667 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.081624 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:58.367366 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.367275 2564 generic.go:356] "Generic (PLEG): container finished" podID="f679ff05-40fa-43c9-b6bc-c71ae339d666" containerID="a37a0ce1fb9e00477bcd81bcbe67cb6d56da19f9a6677d34845d7cc1a097ae73" exitCode=0 Sep 14 03:06:58.367366 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.367316 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8" Sep 14 03:06:58.367618 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.367363 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8" event={"ID":"f679ff05-40fa-43c9-b6bc-c71ae339d666","Type":"ContainerDied","Data":"a37a0ce1fb9e00477bcd81bcbe67cb6d56da19f9a6677d34845d7cc1a097ae73"} Sep 14 03:06:58.367618 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.367416 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8" event={"ID":"f679ff05-40fa-43c9-b6bc-c71ae339d666","Type":"ContainerDied","Data":"d18655722aea5b33efee2e7716c4d3a98d1dc910c1d1e3ba885b09a9520a239e"} Sep 14 03:06:58.367618 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.367442 2564 scope.go:117] "RemoveContainer" containerID="a37a0ce1fb9e00477bcd81bcbe67cb6d56da19f9a6677d34845d7cc1a097ae73" Sep 14 03:06:58.368818 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.368792 2564 generic.go:356] "Generic (PLEG): container finished" podID="1fcb8b38-9617-4d97-89af-f9ead125f4e1" containerID="545db4b69d4cd4eb9ce9c3bdd5eb9e0e11bed3bacaf1a8ecb644a79df133368a" exitCode=0 Sep 14 03:06:58.368920 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.368831 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7" Sep 14 03:06:58.368920 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.368851 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7" event={"ID":"1fcb8b38-9617-4d97-89af-f9ead125f4e1","Type":"ContainerDied","Data":"545db4b69d4cd4eb9ce9c3bdd5eb9e0e11bed3bacaf1a8ecb644a79df133368a"} Sep 14 03:06:58.368920 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.368893 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7" event={"ID":"1fcb8b38-9617-4d97-89af-f9ead125f4e1","Type":"ContainerDied","Data":"f6511e46d6b7a4720cb6ba49b5c9c30f4f35ad62376c96e38725b0d3f98cc58c"} Sep 14 03:06:58.377536 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.377519 2564 scope.go:117] "RemoveContainer" containerID="a37a0ce1fb9e00477bcd81bcbe67cb6d56da19f9a6677d34845d7cc1a097ae73" Sep 14 03:06:58.377872 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:06:58.377848 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a37a0ce1fb9e00477bcd81bcbe67cb6d56da19f9a6677d34845d7cc1a097ae73\": container with ID starting with a37a0ce1fb9e00477bcd81bcbe67cb6d56da19f9a6677d34845d7cc1a097ae73 not found: ID does not exist" containerID="a37a0ce1fb9e00477bcd81bcbe67cb6d56da19f9a6677d34845d7cc1a097ae73" Sep 14 03:06:58.377933 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.377886 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a37a0ce1fb9e00477bcd81bcbe67cb6d56da19f9a6677d34845d7cc1a097ae73"} err="failed to get container status \"a37a0ce1fb9e00477bcd81bcbe67cb6d56da19f9a6677d34845d7cc1a097ae73\": rpc error: code = NotFound desc = could not find container \"a37a0ce1fb9e00477bcd81bcbe67cb6d56da19f9a6677d34845d7cc1a097ae73\": container with ID starting with a37a0ce1fb9e00477bcd81bcbe67cb6d56da19f9a6677d34845d7cc1a097ae73 not found: ID does not exist" Sep 14 03:06:58.377933 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.377914 2564 scope.go:117] "RemoveContainer" containerID="545db4b69d4cd4eb9ce9c3bdd5eb9e0e11bed3bacaf1a8ecb644a79df133368a" Sep 14 03:06:58.386866 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.386842 2564 scope.go:117] "RemoveContainer" containerID="545db4b69d4cd4eb9ce9c3bdd5eb9e0e11bed3bacaf1a8ecb644a79df133368a" Sep 14 03:06:58.387198 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:06:58.387179 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"545db4b69d4cd4eb9ce9c3bdd5eb9e0e11bed3bacaf1a8ecb644a79df133368a\": container with ID starting with 545db4b69d4cd4eb9ce9c3bdd5eb9e0e11bed3bacaf1a8ecb644a79df133368a not found: ID does not exist" containerID="545db4b69d4cd4eb9ce9c3bdd5eb9e0e11bed3bacaf1a8ecb644a79df133368a" Sep 14 03:06:58.387244 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.387207 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"545db4b69d4cd4eb9ce9c3bdd5eb9e0e11bed3bacaf1a8ecb644a79df133368a"} err="failed to get container status \"545db4b69d4cd4eb9ce9c3bdd5eb9e0e11bed3bacaf1a8ecb644a79df133368a\": rpc error: code = NotFound desc = could not find container \"545db4b69d4cd4eb9ce9c3bdd5eb9e0e11bed3bacaf1a8ecb644a79df133368a\": container with ID starting with 545db4b69d4cd4eb9ce9c3bdd5eb9e0e11bed3bacaf1a8ecb644a79df133368a not found: ID does not exist" Sep 14 03:06:58.391913 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.391886 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8"] Sep 14 03:06:58.398066 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.398039 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89bb5623-7858c64ff7-74nr8"] Sep 14 03:06:58.410577 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.410552 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7"] Sep 14 03:06:58.416715 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.416692 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-89bb5623-ccb5678c8-r85c7"] Sep 14 03:06:58.750581 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.750547 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fcb8b38-9617-4d97-89af-f9ead125f4e1" path="/var/lib/kubelet/pods/1fcb8b38-9617-4d97-89af-f9ead125f4e1/volumes" Sep 14 03:06:58.750926 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.750912 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f679ff05-40fa-43c9-b6bc-c71ae339d666" path="/var/lib/kubelet/pods/f679ff05-40fa-43c9-b6bc-c71ae339d666/volumes" Sep 14 03:06:58.826497 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.826447 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:58.834633 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.834604 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx"] Sep 14 03:06:58.834845 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.834819 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" podUID="15512f1a-697b-4b4a-8e5d-d803e3cb1953" containerName="istio-proxy" containerID="cri-o://d33b7b98291b48b17a5d48ec982948c88a06d181d03c9a6b317c0b2303c2f5dc" gracePeriod=30 Sep 14 03:06:58.922085 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:58.922049 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:59.276178 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.276142 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:59.522029 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.521866 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:59.548030 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.547937 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8"] Sep 14 03:06:59.548391 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.548376 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fcb8b38-9617-4d97-89af-f9ead125f4e1" containerName="payload-pre-processing" Sep 14 03:06:59.548442 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.548392 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fcb8b38-9617-4d97-89af-f9ead125f4e1" containerName="payload-pre-processing" Sep 14 03:06:59.548442 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.548420 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f679ff05-40fa-43c9-b6bc-c71ae339d666" containerName="payload-processing" Sep 14 03:06:59.548442 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.548426 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f679ff05-40fa-43c9-b6bc-c71ae339d666" containerName="payload-processing" Sep 14 03:06:59.548582 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.548510 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fcb8b38-9617-4d97-89af-f9ead125f4e1" containerName="payload-pre-processing" Sep 14 03:06:59.548582 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.548522 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f679ff05-40fa-43c9-b6bc-c71ae339d666" containerName="payload-processing" Sep 14 03:06:59.553616 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.553592 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.556185 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.556167 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-853daece-openshift-default-dockercfg-kktfr\"" Sep 14 03:06:59.566674 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.566645 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8"] Sep 14 03:06:59.618890 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.618854 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:59.668909 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.668874 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-data\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.668909 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.668912 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-token\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.669146 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.668951 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-envoy\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.669146 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.669019 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0146a9b6-c656-4cc7-ba7c-b49e54150941-istiod-ca-cert\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.669146 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.669059 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-workload-certs\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.669269 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.669154 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-podinfo\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.669269 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.669172 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-workload-socket\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.669269 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.669189 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-credential-socket\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.669269 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.669206 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdcl9\" (UniqueName: \"kubernetes.io/projected/0146a9b6-c656-4cc7-ba7c-b49e54150941-kube-api-access-sdcl9\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.718832 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.718791 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:06:59.769927 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.769879 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-podinfo\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.769927 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.769923 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-workload-socket\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.769927 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.769938 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-credential-socket\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.770250 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.769956 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdcl9\" (UniqueName: \"kubernetes.io/projected/0146a9b6-c656-4cc7-ba7c-b49e54150941-kube-api-access-sdcl9\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.770250 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.770011 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-data\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.770250 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.770047 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-token\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.770250 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.770080 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-envoy\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.770250 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.770110 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0146a9b6-c656-4cc7-ba7c-b49e54150941-istiod-ca-cert\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.770250 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.770145 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-workload-certs\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.770622 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.770406 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-credential-socket\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.770622 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.770457 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-workload-certs\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.770622 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.770546 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-data\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.770622 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.770610 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-workload-socket\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.770976 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.770956 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0146a9b6-c656-4cc7-ba7c-b49e54150941-istiod-ca-cert\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.772731 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.772701 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-envoy\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.772852 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.772837 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-podinfo\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.780153 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.780118 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-token\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.780288 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.780230 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdcl9\" (UniqueName: \"kubernetes.io/projected/0146a9b6-c656-4cc7-ba7c-b49e54150941-kube-api-access-sdcl9\") pod \"e2e-derive-853daece-openshift-default-664896b9b-xkbg8\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:06:59.864255 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:06:59.864169 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:07:00.143640 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:00.143616 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8"] Sep 14 03:07:00.146198 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:07:00.146174 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0146a9b6_c656_4cc7_ba7c_b49e54150941.slice/crio-f89011bcd540713bad7072220995f4c4af0ab26ef3509fcb996236563ea79695 WatchSource:0}: Error finding container f89011bcd540713bad7072220995f4c4af0ab26ef3509fcb996236563ea79695: Status 404 returned error can't find the container with id f89011bcd540713bad7072220995f4c4af0ab26ef3509fcb996236563ea79695 Sep 14 03:07:00.148839 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:00.148805 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:07:00.148939 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:00.148872 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:07:00.148939 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:00.148901 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:07:00.390894 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:00.390846 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" event={"ID":"0146a9b6-c656-4cc7-ba7c-b49e54150941","Type":"ContainerStarted","Data":"96296bd7709f22bd32d93f84b16a158b176bf139f4d2e43fdbddd6dc10d829c8"} Sep 14 03:07:00.391469 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:00.390912 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" event={"ID":"0146a9b6-c656-4cc7-ba7c-b49e54150941","Type":"ContainerStarted","Data":"f89011bcd540713bad7072220995f4c4af0ab26ef3509fcb996236563ea79695"} Sep 14 03:07:00.445539 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:00.445408 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" podStartSLOduration=1.445388733 podStartE2EDuration="1.445388733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:07:00.442414744 +0000 UTC m=+992.172073861" watchObservedRunningTime="2026-09-14 03:07:00.445388733 +0000 UTC m=+992.175047850" Sep 14 03:07:00.864424 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:00.864326 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:07:01.097015 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:01.096977 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:01.653583 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:01.653541 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:01.787856 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:01.787813 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:01.870223 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:01.870191 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:07:01.884926 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:01.884896 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:02.400796 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:02.400769 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:07:02.401827 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:02.401803 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:07:02.560128 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:02.560096 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:03.673939 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:03.673895 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:03.777763 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:03.777726 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:03.874169 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:03.874137 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:04.087299 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.087276 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:07:04.215155 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.215053 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-credential-socket\") pod \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " Sep 14 03:07:04.215155 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.215129 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-workload-certs\") pod \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " Sep 14 03:07:04.215402 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.215174 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-data\") pod \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " Sep 14 03:07:04.215402 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.215226 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-envoy\") pod \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " Sep 14 03:07:04.215402 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.215268 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-podinfo\") pod \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " Sep 14 03:07:04.215637 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.215299 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istiod-ca-cert\") pod \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " Sep 14 03:07:04.215775 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.215698 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-token\") pod \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " Sep 14 03:07:04.215775 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.215377 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "15512f1a-697b-4b4a-8e5d-d803e3cb1953" (UID: "15512f1a-697b-4b4a-8e5d-d803e3cb1953"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:07:04.215775 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.215424 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "15512f1a-697b-4b4a-8e5d-d803e3cb1953" (UID: "15512f1a-697b-4b4a-8e5d-d803e3cb1953"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:07:04.215775 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.215646 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-data" (OuterVolumeSpecName: "istio-data") pod "15512f1a-697b-4b4a-8e5d-d803e3cb1953" (UID: "15512f1a-697b-4b4a-8e5d-d803e3cb1953"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:07:04.215775 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.215745 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "15512f1a-697b-4b4a-8e5d-d803e3cb1953" (UID: "15512f1a-697b-4b4a-8e5d-d803e3cb1953"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:07:04.215775 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.215743 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xddh2\" (UniqueName: \"kubernetes.io/projected/15512f1a-697b-4b4a-8e5d-d803e3cb1953-kube-api-access-xddh2\") pod \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " Sep 14 03:07:04.216110 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.215814 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-workload-socket\") pod \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\" (UID: \"15512f1a-697b-4b4a-8e5d-d803e3cb1953\") " Sep 14 03:07:04.216255 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.216230 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "15512f1a-697b-4b4a-8e5d-d803e3cb1953" (UID: "15512f1a-697b-4b4a-8e5d-d803e3cb1953"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:07:04.216503 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.216241 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-credential-socket\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:04.216503 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.216280 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-workload-certs\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:04.216503 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.216297 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-data\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:04.216503 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.216310 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istiod-ca-cert\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:04.218003 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.217971 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "15512f1a-697b-4b4a-8e5d-d803e3cb1953" (UID: "15512f1a-697b-4b4a-8e5d-d803e3cb1953"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:07:04.218091 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.218002 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-token" (OuterVolumeSpecName: "istio-token") pod "15512f1a-697b-4b4a-8e5d-d803e3cb1953" (UID: "15512f1a-697b-4b4a-8e5d-d803e3cb1953"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:07:04.218219 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.218198 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "15512f1a-697b-4b4a-8e5d-d803e3cb1953" (UID: "15512f1a-697b-4b4a-8e5d-d803e3cb1953"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 03:07:04.218452 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.218434 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15512f1a-697b-4b4a-8e5d-d803e3cb1953-kube-api-access-xddh2" (OuterVolumeSpecName: "kube-api-access-xddh2") pod "15512f1a-697b-4b4a-8e5d-d803e3cb1953" (UID: "15512f1a-697b-4b4a-8e5d-d803e3cb1953"). InnerVolumeSpecName "kube-api-access-xddh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:07:04.266713 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.266676 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:04.317065 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.317034 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-envoy\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:04.317065 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.317062 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-podinfo\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:04.317065 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.317072 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15512f1a-697b-4b4a-8e5d-d803e3cb1953-istio-token\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:04.317324 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.317080 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xddh2\" (UniqueName: \"kubernetes.io/projected/15512f1a-697b-4b4a-8e5d-d803e3cb1953-kube-api-access-xddh2\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:04.317324 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.317092 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15512f1a-697b-4b4a-8e5d-d803e3cb1953-workload-socket\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:04.358991 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.358955 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk"] Sep 14 03:07:04.359379 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.359366 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15512f1a-697b-4b4a-8e5d-d803e3cb1953" containerName="istio-proxy" Sep 14 03:07:04.359431 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.359380 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="15512f1a-697b-4b4a-8e5d-d803e3cb1953" containerName="istio-proxy" Sep 14 03:07:04.359431 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.359395 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15512f1a-697b-4b4a-8e5d-d803e3cb1953" containerName="istio-proxy" Sep 14 03:07:04.359431 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.359401 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="15512f1a-697b-4b4a-8e5d-d803e3cb1953" containerName="istio-proxy" Sep 14 03:07:04.359543 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.359468 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="15512f1a-697b-4b4a-8e5d-d803e3cb1953" containerName="istio-proxy" Sep 14 03:07:04.359543 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.359492 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="15512f1a-697b-4b4a-8e5d-d803e3cb1953" containerName="istio-proxy" Sep 14 03:07:04.363702 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.363676 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:04.366565 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.366538 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk" Sep 14 03:07:04.370229 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.370207 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-853daece\"" Sep 14 03:07:04.380930 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.380904 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk"] Sep 14 03:07:04.394445 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.394408 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt"] Sep 14 03:07:04.399805 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.399775 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt" Sep 14 03:07:04.402437 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.402412 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-853daece-dockercfg-jcbmf\"" Sep 14 03:07:04.408325 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.408296 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt"] Sep 14 03:07:04.414262 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.414233 2564 generic.go:356] "Generic (PLEG): container finished" podID="15512f1a-697b-4b4a-8e5d-d803e3cb1953" containerID="d33b7b98291b48b17a5d48ec982948c88a06d181d03c9a6b317c0b2303c2f5dc" exitCode=0 Sep 14 03:07:04.414382 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.414311 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" event={"ID":"15512f1a-697b-4b4a-8e5d-d803e3cb1953","Type":"ContainerDied","Data":"d33b7b98291b48b17a5d48ec982948c88a06d181d03c9a6b317c0b2303c2f5dc"} Sep 14 03:07:04.414382 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.414368 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" event={"ID":"15512f1a-697b-4b4a-8e5d-d803e3cb1953","Type":"ContainerDied","Data":"b3c80507e8b73629ebbf90d34e375dd5f11a2f19982a1a82e7a067aa3010d853"} Sep 14 03:07:04.414455 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.414315 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx" Sep 14 03:07:04.414455 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.414398 2564 scope.go:117] "RemoveContainer" containerID="d33b7b98291b48b17a5d48ec982948c88a06d181d03c9a6b317c0b2303c2f5dc" Sep 14 03:07:04.426202 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.426176 2564 scope.go:117] "RemoveContainer" containerID="8f26373da36c84960542073d35afce22e9dafbb1a9ba363eafbafcb5f676bdfb" Sep 14 03:07:04.435051 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.435032 2564 scope.go:117] "RemoveContainer" containerID="d33b7b98291b48b17a5d48ec982948c88a06d181d03c9a6b317c0b2303c2f5dc" Sep 14 03:07:04.435405 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:07:04.435383 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d33b7b98291b48b17a5d48ec982948c88a06d181d03c9a6b317c0b2303c2f5dc\": container with ID starting with d33b7b98291b48b17a5d48ec982948c88a06d181d03c9a6b317c0b2303c2f5dc not found: ID does not exist" containerID="d33b7b98291b48b17a5d48ec982948c88a06d181d03c9a6b317c0b2303c2f5dc" Sep 14 03:07:04.435456 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.435414 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d33b7b98291b48b17a5d48ec982948c88a06d181d03c9a6b317c0b2303c2f5dc"} err="failed to get container status \"d33b7b98291b48b17a5d48ec982948c88a06d181d03c9a6b317c0b2303c2f5dc\": rpc error: code = NotFound desc = could not find container \"d33b7b98291b48b17a5d48ec982948c88a06d181d03c9a6b317c0b2303c2f5dc\": container with ID starting with d33b7b98291b48b17a5d48ec982948c88a06d181d03c9a6b317c0b2303c2f5dc not found: ID does not exist" Sep 14 03:07:04.435456 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.435435 2564 scope.go:117] "RemoveContainer" containerID="8f26373da36c84960542073d35afce22e9dafbb1a9ba363eafbafcb5f676bdfb" Sep 14 03:07:04.435750 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:07:04.435732 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f26373da36c84960542073d35afce22e9dafbb1a9ba363eafbafcb5f676bdfb\": container with ID starting with 8f26373da36c84960542073d35afce22e9dafbb1a9ba363eafbafcb5f676bdfb not found: ID does not exist" containerID="8f26373da36c84960542073d35afce22e9dafbb1a9ba363eafbafcb5f676bdfb" Sep 14 03:07:04.435794 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.435756 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f26373da36c84960542073d35afce22e9dafbb1a9ba363eafbafcb5f676bdfb"} err="failed to get container status \"8f26373da36c84960542073d35afce22e9dafbb1a9ba363eafbafcb5f676bdfb\": rpc error: code = NotFound desc = could not find container \"8f26373da36c84960542073d35afce22e9dafbb1a9ba363eafbafcb5f676bdfb\": container with ID starting with 8f26373da36c84960542073d35afce22e9dafbb1a9ba363eafbafcb5f676bdfb not found: ID does not exist" Sep 14 03:07:04.440132 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.440103 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx"] Sep 14 03:07:04.443043 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.443020 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-89bb5623-openshift-default-7b9df9846b-6g8hx"] Sep 14 03:07:04.519243 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.519133 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppz4k\" (UniqueName: \"kubernetes.io/projected/2dd13925-79e3-47c8-95df-c5cbf08e1f14-kube-api-access-ppz4k\") pod \"payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt\" (UID: \"2dd13925-79e3-47c8-95df-c5cbf08e1f14\") " pod="openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt" Sep 14 03:07:04.519428 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.519282 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81882939-af3b-4659-909b-8463416737dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk\" (UID: \"81882939-af3b-4659-909b-8463416737dc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk" Sep 14 03:07:04.519428 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.519331 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8bsc\" (UniqueName: \"kubernetes.io/projected/81882939-af3b-4659-909b-8463416737dc-kube-api-access-t8bsc\") pod \"payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk\" (UID: \"81882939-af3b-4659-909b-8463416737dc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk" Sep 14 03:07:04.519428 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.519388 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd13925-79e3-47c8-95df-c5cbf08e1f14-plugins-config-volume\") pod \"payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt\" (UID: \"2dd13925-79e3-47c8-95df-c5cbf08e1f14\") " pod="openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt" Sep 14 03:07:04.620746 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.620712 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd13925-79e3-47c8-95df-c5cbf08e1f14-plugins-config-volume\") pod \"payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt\" (UID: \"2dd13925-79e3-47c8-95df-c5cbf08e1f14\") " pod="openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt" Sep 14 03:07:04.620909 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.620819 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppz4k\" (UniqueName: \"kubernetes.io/projected/2dd13925-79e3-47c8-95df-c5cbf08e1f14-kube-api-access-ppz4k\") pod \"payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt\" (UID: \"2dd13925-79e3-47c8-95df-c5cbf08e1f14\") " pod="openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt" Sep 14 03:07:04.620909 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.620870 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81882939-af3b-4659-909b-8463416737dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk\" (UID: \"81882939-af3b-4659-909b-8463416737dc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk" Sep 14 03:07:04.620909 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.620900 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8bsc\" (UniqueName: \"kubernetes.io/projected/81882939-af3b-4659-909b-8463416737dc-kube-api-access-t8bsc\") pod \"payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk\" (UID: \"81882939-af3b-4659-909b-8463416737dc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk" Sep 14 03:07:04.621367 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.621342 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd13925-79e3-47c8-95df-c5cbf08e1f14-plugins-config-volume\") pod \"payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt\" (UID: \"2dd13925-79e3-47c8-95df-c5cbf08e1f14\") " pod="openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt" Sep 14 03:07:04.621612 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.621591 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81882939-af3b-4659-909b-8463416737dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk\" (UID: \"81882939-af3b-4659-909b-8463416737dc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk" Sep 14 03:07:04.630509 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.630452 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppz4k\" (UniqueName: \"kubernetes.io/projected/2dd13925-79e3-47c8-95df-c5cbf08e1f14-kube-api-access-ppz4k\") pod \"payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt\" (UID: \"2dd13925-79e3-47c8-95df-c5cbf08e1f14\") " pod="openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt" Sep 14 03:07:04.630509 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.630496 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8bsc\" (UniqueName: \"kubernetes.io/projected/81882939-af3b-4659-909b-8463416737dc-kube-api-access-t8bsc\") pod \"payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk\" (UID: \"81882939-af3b-4659-909b-8463416737dc\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk" Sep 14 03:07:04.679298 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.679253 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk" Sep 14 03:07:04.714428 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.714390 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt" Sep 14 03:07:04.755489 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.755404 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15512f1a-697b-4b4a-8e5d-d803e3cb1953" path="/var/lib/kubelet/pods/15512f1a-697b-4b4a-8e5d-d803e3cb1953/volumes" Sep 14 03:07:04.954609 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:04.954581 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk"] Sep 14 03:07:04.956338 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:07:04.956309 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81882939_af3b_4659_909b_8463416737dc.slice/crio-15063df7d66a6f7a25bb6564416a3942327928cdd57fe39e1382687641377efc WatchSource:0}: Error finding container 15063df7d66a6f7a25bb6564416a3942327928cdd57fe39e1382687641377efc: Status 404 returned error can't find the container with id 15063df7d66a6f7a25bb6564416a3942327928cdd57fe39e1382687641377efc Sep 14 03:07:05.008674 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:05.008649 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt"] Sep 14 03:07:05.010959 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:07:05.010931 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd13925_79e3_47c8_95df_c5cbf08e1f14.slice/crio-2d84befb44bdb8bc90a2d0295319ec32c13b5c05e0a8e8ae39f736c78d41c593 WatchSource:0}: Error finding container 2d84befb44bdb8bc90a2d0295319ec32c13b5c05e0a8e8ae39f736c78d41c593: Status 404 returned error can't find the container with id 2d84befb44bdb8bc90a2d0295319ec32c13b5c05e0a8e8ae39f736c78d41c593 Sep 14 03:07:05.423253 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:05.423214 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk" event={"ID":"81882939-af3b-4659-909b-8463416737dc","Type":"ContainerStarted","Data":"ee92dc3b4fa709add011b8d322dcf7cd476e68eaff577a3c72d9a02e941930fa"} Sep 14 03:07:05.423465 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:05.423285 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk" event={"ID":"81882939-af3b-4659-909b-8463416737dc","Type":"ContainerStarted","Data":"15063df7d66a6f7a25bb6564416a3942327928cdd57fe39e1382687641377efc"} Sep 14 03:07:05.423465 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:05.423340 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk" Sep 14 03:07:05.426002 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:05.425963 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt" event={"ID":"2dd13925-79e3-47c8-95df-c5cbf08e1f14","Type":"ContainerStarted","Data":"8087a56355b519f0e2adb40a0c9e15de104cb5e600b2fa532d6d771bb8f18ec1"} Sep 14 03:07:05.426184 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:05.426016 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt" event={"ID":"2dd13925-79e3-47c8-95df-c5cbf08e1f14","Type":"ContainerStarted","Data":"2d84befb44bdb8bc90a2d0295319ec32c13b5c05e0a8e8ae39f736c78d41c593"} Sep 14 03:07:05.426184 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:05.426043 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt" Sep 14 03:07:05.441133 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:05.441078 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk" podStartSLOduration=1.441062902 podStartE2EDuration="1.441062902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:07:05.438701486 +0000 UTC m=+997.168360604" watchObservedRunningTime="2026-09-14 03:07:05.441062902 +0000 UTC m=+997.170722018" Sep 14 03:07:05.457108 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:05.457054 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt" podStartSLOduration=1.457038133 podStartE2EDuration="1.457038133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:07:05.455843992 +0000 UTC m=+997.185503106" watchObservedRunningTime="2026-09-14 03:07:05.457038133 +0000 UTC m=+997.186697249" Sep 14 03:07:13.767834 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:13.767796 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:14.695503 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:14.695130 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:15.018837 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:15.018751 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:15.822845 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:15.822811 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:16.437947 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:16.437859 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk" Sep 14 03:07:16.438342 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:16.438225 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt" Sep 14 03:07:16.621098 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:16.621050 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:17.382854 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:17.382799 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:18.128788 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:18.128747 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:18.851000 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:18.850966 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:19.561164 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:19.561121 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:20.325995 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:20.325948 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:20.878012 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:20.877971 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:21.533891 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:21.533858 2564 generic.go:356] "Generic (PLEG): container finished" podID="0146a9b6-c656-4cc7-ba7c-b49e54150941" containerID="96296bd7709f22bd32d93f84b16a158b176bf139f4d2e43fdbddd6dc10d829c8" exitCode=0 Sep 14 03:07:21.534119 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:21.533907 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" event={"ID":"0146a9b6-c656-4cc7-ba7c-b49e54150941","Type":"ContainerDied","Data":"96296bd7709f22bd32d93f84b16a158b176bf139f4d2e43fdbddd6dc10d829c8"} Sep 14 03:07:21.534328 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:21.534313 2564 scope.go:117] "RemoveContainer" containerID="96296bd7709f22bd32d93f84b16a158b176bf139f4d2e43fdbddd6dc10d829c8" Sep 14 03:07:21.535402 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:21.535373 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:07:21.535466 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:21.535444 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:07:21.535531 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:21.535494 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:07:21.729064 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:21.729023 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:22.425836 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:22.425799 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:22.541672 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:22.541636 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" event={"ID":"0146a9b6-c656-4cc7-ba7c-b49e54150941","Type":"ContainerStarted","Data":"ce3177f7f1741daae4207fbb1f113367b20ff0348b314121a2c0a0cf3382651d"} Sep 14 03:07:22.865285 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:22.865195 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:07:22.865458 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:22.865437 2564 patch_prober.go:28] interesting pod/e2e-derive-853daece-openshift-default-664896b9b-xkbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 14 03:07:22.865555 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:22.865512 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" podUID="0146a9b6-c656-4cc7-ba7c-b49e54150941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 14 03:07:23.228452 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:23.228417 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:23.865282 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:23.865238 2564 patch_prober.go:28] interesting pod/e2e-derive-853daece-openshift-default-664896b9b-xkbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 14 03:07:23.865719 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:23.865300 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" podUID="0146a9b6-c656-4cc7-ba7c-b49e54150941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 14 03:07:23.991005 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:23.990968 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:24.820385 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:24.820351 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:24.865629 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:24.865594 2564 patch_prober.go:28] interesting pod/e2e-derive-853daece-openshift-default-664896b9b-xkbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 14 03:07:24.866050 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:24.865648 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" podUID="0146a9b6-c656-4cc7-ba7c-b49e54150941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 14 03:07:25.453210 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:25.453170 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:25.532440 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:25.532406 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx"] Sep 14 03:07:25.532792 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:25.532738 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx" podUID="31915719-a15a-449f-814f-84f35762a374" containerName="payload-processing" containerID="cri-o://35c83e3fa3fb031bad3fc82bfc6f0b1ee7af805cdca111b35576b9d3ae1f2050" gracePeriod=30 Sep 14 03:07:25.653878 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:25.653816 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:25.749251 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:25.749167 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:25.831752 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:25.831729 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx" Sep 14 03:07:25.846106 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:25.846073 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:25.865240 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:25.865210 2564 patch_prober.go:28] interesting pod/e2e-derive-853daece-openshift-default-664896b9b-xkbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 14 03:07:25.865370 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:25.865264 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" podUID="0146a9b6-c656-4cc7-ba7c-b49e54150941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 14 03:07:25.936845 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:25.936817 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31915719-a15a-449f-814f-84f35762a374-plugins-config-volume\") pod \"31915719-a15a-449f-814f-84f35762a374\" (UID: \"31915719-a15a-449f-814f-84f35762a374\") " Sep 14 03:07:25.937196 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:25.936896 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sz9pf\" (UniqueName: \"kubernetes.io/projected/31915719-a15a-449f-814f-84f35762a374-kube-api-access-sz9pf\") pod \"31915719-a15a-449f-814f-84f35762a374\" (UID: \"31915719-a15a-449f-814f-84f35762a374\") " Sep 14 03:07:25.937196 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:25.937156 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31915719-a15a-449f-814f-84f35762a374-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "31915719-a15a-449f-814f-84f35762a374" (UID: "31915719-a15a-449f-814f-84f35762a374"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:07:25.938918 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:25.938899 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31915719-a15a-449f-814f-84f35762a374-kube-api-access-sz9pf" (OuterVolumeSpecName: "kube-api-access-sz9pf") pod "31915719-a15a-449f-814f-84f35762a374" (UID: "31915719-a15a-449f-814f-84f35762a374"). InnerVolumeSpecName "kube-api-access-sz9pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:07:26.038111 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:26.038024 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31915719-a15a-449f-814f-84f35762a374-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:26.038111 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:26.038053 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz9pf\" (UniqueName: \"kubernetes.io/projected/31915719-a15a-449f-814f-84f35762a374-kube-api-access-sz9pf\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:26.156000 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:26.155963 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:26.253747 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:26.253701 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:26.568519 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:26.568462 2564 generic.go:356] "Generic (PLEG): container finished" podID="31915719-a15a-449f-814f-84f35762a374" containerID="35c83e3fa3fb031bad3fc82bfc6f0b1ee7af805cdca111b35576b9d3ae1f2050" exitCode=0 Sep 14 03:07:26.568698 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:26.568528 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx" Sep 14 03:07:26.568698 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:26.568547 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx" event={"ID":"31915719-a15a-449f-814f-84f35762a374","Type":"ContainerDied","Data":"35c83e3fa3fb031bad3fc82bfc6f0b1ee7af805cdca111b35576b9d3ae1f2050"} Sep 14 03:07:26.568698 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:26.568620 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx" event={"ID":"31915719-a15a-449f-814f-84f35762a374","Type":"ContainerDied","Data":"3e4953e62daffabe24e8409b38d428422a70f1fde42bf4850ad785b3976e2b74"} Sep 14 03:07:26.568698 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:26.568642 2564 scope.go:117] "RemoveContainer" containerID="35c83e3fa3fb031bad3fc82bfc6f0b1ee7af805cdca111b35576b9d3ae1f2050" Sep 14 03:07:26.595688 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:26.595657 2564 scope.go:117] "RemoveContainer" containerID="35c83e3fa3fb031bad3fc82bfc6f0b1ee7af805cdca111b35576b9d3ae1f2050" Sep 14 03:07:26.601681 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:07:26.601643 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35c83e3fa3fb031bad3fc82bfc6f0b1ee7af805cdca111b35576b9d3ae1f2050\": container with ID starting with 35c83e3fa3fb031bad3fc82bfc6f0b1ee7af805cdca111b35576b9d3ae1f2050 not found: ID does not exist" containerID="35c83e3fa3fb031bad3fc82bfc6f0b1ee7af805cdca111b35576b9d3ae1f2050" Sep 14 03:07:26.601810 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:26.601693 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35c83e3fa3fb031bad3fc82bfc6f0b1ee7af805cdca111b35576b9d3ae1f2050"} err="failed to get container status \"35c83e3fa3fb031bad3fc82bfc6f0b1ee7af805cdca111b35576b9d3ae1f2050\": rpc error: code = NotFound desc = could not find container \"35c83e3fa3fb031bad3fc82bfc6f0b1ee7af805cdca111b35576b9d3ae1f2050\": container with ID starting with 35c83e3fa3fb031bad3fc82bfc6f0b1ee7af805cdca111b35576b9d3ae1f2050 not found: ID does not exist" Sep 14 03:07:26.618508 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:26.618465 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx"] Sep 14 03:07:26.623094 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:26.623064 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e467de-5bdbf8f9-m9pvx"] Sep 14 03:07:26.751114 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:26.751079 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31915719-a15a-449f-814f-84f35762a374" path="/var/lib/kubelet/pods/31915719-a15a-449f-814f-84f35762a374/volumes" Sep 14 03:07:26.864951 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:26.864858 2564 patch_prober.go:28] interesting pod/e2e-derive-853daece-openshift-default-664896b9b-xkbg8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Sep 14 03:07:26.864951 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:26.864921 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" podUID="0146a9b6-c656-4cc7-ba7c-b49e54150941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Sep 14 03:07:26.935129 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:26.935080 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:27.739946 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:27.739906 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:27.870100 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:27.870069 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:07:27.870488 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:27.870444 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:07:27.871362 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:27.871340 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:07:32.053339 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.053299 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:32.080524 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.080487 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt"] Sep 14 03:07:32.080819 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.080774 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt" podUID="2dd13925-79e3-47c8-95df-c5cbf08e1f14" containerName="payload-processing" containerID="cri-o://8087a56355b519f0e2adb40a0c9e15de104cb5e600b2fa532d6d771bb8f18ec1" gracePeriod=30 Sep 14 03:07:32.096708 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.096674 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk"] Sep 14 03:07:32.096931 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.096906 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk" podUID="81882939-af3b-4659-909b-8463416737dc" containerName="payload-pre-processing" containerID="cri-o://ee92dc3b4fa709add011b8d322dcf7cd476e68eaff577a3c72d9a02e941930fa" gracePeriod=30 Sep 14 03:07:32.157273 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.157233 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:32.258516 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.258438 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:32.343261 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.343219 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:32.343565 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.343549 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt" Sep 14 03:07:32.373785 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.373758 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk" Sep 14 03:07:32.400879 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.400840 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppz4k\" (UniqueName: \"kubernetes.io/projected/2dd13925-79e3-47c8-95df-c5cbf08e1f14-kube-api-access-ppz4k\") pod \"2dd13925-79e3-47c8-95df-c5cbf08e1f14\" (UID: \"2dd13925-79e3-47c8-95df-c5cbf08e1f14\") " Sep 14 03:07:32.401057 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.400908 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd13925-79e3-47c8-95df-c5cbf08e1f14-plugins-config-volume\") pod \"2dd13925-79e3-47c8-95df-c5cbf08e1f14\" (UID: \"2dd13925-79e3-47c8-95df-c5cbf08e1f14\") " Sep 14 03:07:32.401294 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.401259 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd13925-79e3-47c8-95df-c5cbf08e1f14-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2dd13925-79e3-47c8-95df-c5cbf08e1f14" (UID: "2dd13925-79e3-47c8-95df-c5cbf08e1f14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:07:32.403647 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.403623 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd13925-79e3-47c8-95df-c5cbf08e1f14-kube-api-access-ppz4k" (OuterVolumeSpecName: "kube-api-access-ppz4k") pod "2dd13925-79e3-47c8-95df-c5cbf08e1f14" (UID: "2dd13925-79e3-47c8-95df-c5cbf08e1f14"). InnerVolumeSpecName "kube-api-access-ppz4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:07:32.468122 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.468077 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8"] Sep 14 03:07:32.468326 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.468276 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" podUID="0146a9b6-c656-4cc7-ba7c-b49e54150941" containerName="istio-proxy" containerID="cri-o://ce3177f7f1741daae4207fbb1f113367b20ff0348b314121a2c0a0cf3382651d" gracePeriod=30 Sep 14 03:07:32.485776 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.485733 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:32.502542 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.502507 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8bsc\" (UniqueName: \"kubernetes.io/projected/81882939-af3b-4659-909b-8463416737dc-kube-api-access-t8bsc\") pod \"81882939-af3b-4659-909b-8463416737dc\" (UID: \"81882939-af3b-4659-909b-8463416737dc\") " Sep 14 03:07:32.502737 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.502582 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81882939-af3b-4659-909b-8463416737dc-plugins-config-volume\") pod \"81882939-af3b-4659-909b-8463416737dc\" (UID: \"81882939-af3b-4659-909b-8463416737dc\") " Sep 14 03:07:32.502947 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.502923 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81882939-af3b-4659-909b-8463416737dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "81882939-af3b-4659-909b-8463416737dc" (UID: "81882939-af3b-4659-909b-8463416737dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:07:32.503014 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.502996 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81882939-af3b-4659-909b-8463416737dc-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:32.503014 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.503008 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppz4k\" (UniqueName: \"kubernetes.io/projected/2dd13925-79e3-47c8-95df-c5cbf08e1f14-kube-api-access-ppz4k\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:32.503089 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.503021 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd13925-79e3-47c8-95df-c5cbf08e1f14-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:32.504931 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.504907 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81882939-af3b-4659-909b-8463416737dc-kube-api-access-t8bsc" (OuterVolumeSpecName: "kube-api-access-t8bsc") pod "81882939-af3b-4659-909b-8463416737dc" (UID: "81882939-af3b-4659-909b-8463416737dc"). InnerVolumeSpecName "kube-api-access-t8bsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:07:32.603079 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.602996 2564 generic.go:356] "Generic (PLEG): container finished" podID="2dd13925-79e3-47c8-95df-c5cbf08e1f14" containerID="8087a56355b519f0e2adb40a0c9e15de104cb5e600b2fa532d6d771bb8f18ec1" exitCode=0 Sep 14 03:07:32.603079 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.603047 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt" Sep 14 03:07:32.603305 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.603078 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt" event={"ID":"2dd13925-79e3-47c8-95df-c5cbf08e1f14","Type":"ContainerDied","Data":"8087a56355b519f0e2adb40a0c9e15de104cb5e600b2fa532d6d771bb8f18ec1"} Sep 14 03:07:32.603305 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.603134 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt" event={"ID":"2dd13925-79e3-47c8-95df-c5cbf08e1f14","Type":"ContainerDied","Data":"2d84befb44bdb8bc90a2d0295319ec32c13b5c05e0a8e8ae39f736c78d41c593"} Sep 14 03:07:32.603305 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.603162 2564 scope.go:117] "RemoveContainer" containerID="8087a56355b519f0e2adb40a0c9e15de104cb5e600b2fa532d6d771bb8f18ec1" Sep 14 03:07:32.603613 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.603588 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8bsc\" (UniqueName: \"kubernetes.io/projected/81882939-af3b-4659-909b-8463416737dc-kube-api-access-t8bsc\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:32.604567 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.604533 2564 generic.go:356] "Generic (PLEG): container finished" podID="81882939-af3b-4659-909b-8463416737dc" containerID="ee92dc3b4fa709add011b8d322dcf7cd476e68eaff577a3c72d9a02e941930fa" exitCode=0 Sep 14 03:07:32.604664 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.604577 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk" Sep 14 03:07:32.604664 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.604606 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk" event={"ID":"81882939-af3b-4659-909b-8463416737dc","Type":"ContainerDied","Data":"ee92dc3b4fa709add011b8d322dcf7cd476e68eaff577a3c72d9a02e941930fa"} Sep 14 03:07:32.604664 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.604648 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk" event={"ID":"81882939-af3b-4659-909b-8463416737dc","Type":"ContainerDied","Data":"15063df7d66a6f7a25bb6564416a3942327928cdd57fe39e1382687641377efc"} Sep 14 03:07:32.612107 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.612005 2564 scope.go:117] "RemoveContainer" containerID="8087a56355b519f0e2adb40a0c9e15de104cb5e600b2fa532d6d771bb8f18ec1" Sep 14 03:07:32.612323 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:07:32.612296 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8087a56355b519f0e2adb40a0c9e15de104cb5e600b2fa532d6d771bb8f18ec1\": container with ID starting with 8087a56355b519f0e2adb40a0c9e15de104cb5e600b2fa532d6d771bb8f18ec1 not found: ID does not exist" containerID="8087a56355b519f0e2adb40a0c9e15de104cb5e600b2fa532d6d771bb8f18ec1" Sep 14 03:07:32.612448 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.612334 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8087a56355b519f0e2adb40a0c9e15de104cb5e600b2fa532d6d771bb8f18ec1"} err="failed to get container status \"8087a56355b519f0e2adb40a0c9e15de104cb5e600b2fa532d6d771bb8f18ec1\": rpc error: code = NotFound desc = could not find container \"8087a56355b519f0e2adb40a0c9e15de104cb5e600b2fa532d6d771bb8f18ec1\": container with ID starting with 8087a56355b519f0e2adb40a0c9e15de104cb5e600b2fa532d6d771bb8f18ec1 not found: ID does not exist" Sep 14 03:07:32.612448 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.612352 2564 scope.go:117] "RemoveContainer" containerID="ee92dc3b4fa709add011b8d322dcf7cd476e68eaff577a3c72d9a02e941930fa" Sep 14 03:07:32.621317 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.621300 2564 scope.go:117] "RemoveContainer" containerID="ee92dc3b4fa709add011b8d322dcf7cd476e68eaff577a3c72d9a02e941930fa" Sep 14 03:07:32.621616 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:07:32.621595 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee92dc3b4fa709add011b8d322dcf7cd476e68eaff577a3c72d9a02e941930fa\": container with ID starting with ee92dc3b4fa709add011b8d322dcf7cd476e68eaff577a3c72d9a02e941930fa not found: ID does not exist" containerID="ee92dc3b4fa709add011b8d322dcf7cd476e68eaff577a3c72d9a02e941930fa" Sep 14 03:07:32.621680 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.621622 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee92dc3b4fa709add011b8d322dcf7cd476e68eaff577a3c72d9a02e941930fa"} err="failed to get container status \"ee92dc3b4fa709add011b8d322dcf7cd476e68eaff577a3c72d9a02e941930fa\": rpc error: code = NotFound desc = could not find container \"ee92dc3b4fa709add011b8d322dcf7cd476e68eaff577a3c72d9a02e941930fa\": container with ID starting with ee92dc3b4fa709add011b8d322dcf7cd476e68eaff577a3c72d9a02e941930fa not found: ID does not exist" Sep 14 03:07:32.647129 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.647096 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk"] Sep 14 03:07:32.651674 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.651650 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-853daece-94f67c57d-z75lk"] Sep 14 03:07:32.663721 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.663698 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt"] Sep 14 03:07:32.668107 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.668085 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-853daece-647bf6f75c-tqbnt"] Sep 14 03:07:32.750654 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.750627 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dd13925-79e3-47c8-95df-c5cbf08e1f14" path="/var/lib/kubelet/pods/2dd13925-79e3-47c8-95df-c5cbf08e1f14/volumes" Sep 14 03:07:32.750995 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:32.750973 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81882939-af3b-4659-909b-8463416737dc" path="/var/lib/kubelet/pods/81882939-af3b-4659-909b-8463416737dc/volumes" Sep 14 03:07:33.024255 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:33.024221 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:33.224725 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:33.224692 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:33.332150 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:33.332054 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:34.053652 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:34.053615 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:34.438062 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:34.438028 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:34.538226 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:34.538191 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:34.720856 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:34.720766 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:34.815195 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:34.815163 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:35.921146 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:35.921115 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:36.578516 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:36.578461 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:37.824356 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:37.824330 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:07:37.952773 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:37.952733 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-token\") pod \"0146a9b6-c656-4cc7-ba7c-b49e54150941\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " Sep 14 03:07:37.952941 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:37.952794 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-credential-socket\") pod \"0146a9b6-c656-4cc7-ba7c-b49e54150941\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " Sep 14 03:07:37.952941 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:37.952822 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-podinfo\") pod \"0146a9b6-c656-4cc7-ba7c-b49e54150941\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " Sep 14 03:07:37.952941 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:37.952862 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-data\") pod \"0146a9b6-c656-4cc7-ba7c-b49e54150941\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " Sep 14 03:07:37.952941 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:37.952905 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-workload-socket\") pod \"0146a9b6-c656-4cc7-ba7c-b49e54150941\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " Sep 14 03:07:37.952941 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:37.952935 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-workload-certs\") pod \"0146a9b6-c656-4cc7-ba7c-b49e54150941\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " Sep 14 03:07:37.953211 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:37.952954 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0146a9b6-c656-4cc7-ba7c-b49e54150941-istiod-ca-cert\") pod \"0146a9b6-c656-4cc7-ba7c-b49e54150941\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " Sep 14 03:07:37.953211 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:37.953000 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdcl9\" (UniqueName: \"kubernetes.io/projected/0146a9b6-c656-4cc7-ba7c-b49e54150941-kube-api-access-sdcl9\") pod \"0146a9b6-c656-4cc7-ba7c-b49e54150941\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " Sep 14 03:07:37.953211 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:37.953032 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-envoy\") pod \"0146a9b6-c656-4cc7-ba7c-b49e54150941\" (UID: \"0146a9b6-c656-4cc7-ba7c-b49e54150941\") " Sep 14 03:07:37.953374 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:37.953131 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0146a9b6-c656-4cc7-ba7c-b49e54150941" (UID: "0146a9b6-c656-4cc7-ba7c-b49e54150941"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:07:37.953374 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:37.953244 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-data" (OuterVolumeSpecName: "istio-data") pod "0146a9b6-c656-4cc7-ba7c-b49e54150941" (UID: "0146a9b6-c656-4cc7-ba7c-b49e54150941"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:07:37.953374 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:37.953245 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0146a9b6-c656-4cc7-ba7c-b49e54150941" (UID: "0146a9b6-c656-4cc7-ba7c-b49e54150941"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:07:37.953374 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:37.953267 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0146a9b6-c656-4cc7-ba7c-b49e54150941" (UID: "0146a9b6-c656-4cc7-ba7c-b49e54150941"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:07:37.953593 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:37.953422 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0146a9b6-c656-4cc7-ba7c-b49e54150941-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0146a9b6-c656-4cc7-ba7c-b49e54150941" (UID: "0146a9b6-c656-4cc7-ba7c-b49e54150941"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:07:37.955422 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:37.955400 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0146a9b6-c656-4cc7-ba7c-b49e54150941" (UID: "0146a9b6-c656-4cc7-ba7c-b49e54150941"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:07:37.955557 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:37.955446 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0146a9b6-c656-4cc7-ba7c-b49e54150941-kube-api-access-sdcl9" (OuterVolumeSpecName: "kube-api-access-sdcl9") pod "0146a9b6-c656-4cc7-ba7c-b49e54150941" (UID: "0146a9b6-c656-4cc7-ba7c-b49e54150941"). InnerVolumeSpecName "kube-api-access-sdcl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:07:37.955615 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:37.955590 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0146a9b6-c656-4cc7-ba7c-b49e54150941" (UID: "0146a9b6-c656-4cc7-ba7c-b49e54150941"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 03:07:37.955615 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:37.955607 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-token" (OuterVolumeSpecName: "istio-token") pod "0146a9b6-c656-4cc7-ba7c-b49e54150941" (UID: "0146a9b6-c656-4cc7-ba7c-b49e54150941"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:07:38.053770 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:38.053735 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-credential-socket\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:38.053770 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:38.053763 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-podinfo\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:38.053770 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:38.053774 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-data\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:38.054000 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:38.053783 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-workload-socket\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:38.054000 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:38.053791 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-workload-certs\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:38.054000 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:38.053799 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0146a9b6-c656-4cc7-ba7c-b49e54150941-istiod-ca-cert\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:38.054000 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:38.053807 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdcl9\" (UniqueName: \"kubernetes.io/projected/0146a9b6-c656-4cc7-ba7c-b49e54150941-kube-api-access-sdcl9\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:38.054000 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:38.053815 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-envoy\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:38.054000 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:38.053822 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0146a9b6-c656-4cc7-ba7c-b49e54150941-istio-token\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:38.639445 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:38.639409 2564 generic.go:356] "Generic (PLEG): container finished" podID="0146a9b6-c656-4cc7-ba7c-b49e54150941" containerID="ce3177f7f1741daae4207fbb1f113367b20ff0348b314121a2c0a0cf3382651d" exitCode=0 Sep 14 03:07:38.639703 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:38.639461 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" Sep 14 03:07:38.639703 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:38.639521 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" event={"ID":"0146a9b6-c656-4cc7-ba7c-b49e54150941","Type":"ContainerDied","Data":"ce3177f7f1741daae4207fbb1f113367b20ff0348b314121a2c0a0cf3382651d"} Sep 14 03:07:38.639703 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:38.639577 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8" event={"ID":"0146a9b6-c656-4cc7-ba7c-b49e54150941","Type":"ContainerDied","Data":"f89011bcd540713bad7072220995f4c4af0ab26ef3509fcb996236563ea79695"} Sep 14 03:07:38.639703 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:38.639603 2564 scope.go:117] "RemoveContainer" containerID="ce3177f7f1741daae4207fbb1f113367b20ff0348b314121a2c0a0cf3382651d" Sep 14 03:07:38.649332 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:38.649315 2564 scope.go:117] "RemoveContainer" containerID="96296bd7709f22bd32d93f84b16a158b176bf139f4d2e43fdbddd6dc10d829c8" Sep 14 03:07:38.657498 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:38.657455 2564 scope.go:117] "RemoveContainer" containerID="ce3177f7f1741daae4207fbb1f113367b20ff0348b314121a2c0a0cf3382651d" Sep 14 03:07:38.657779 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:07:38.657756 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce3177f7f1741daae4207fbb1f113367b20ff0348b314121a2c0a0cf3382651d\": container with ID starting with ce3177f7f1741daae4207fbb1f113367b20ff0348b314121a2c0a0cf3382651d not found: ID does not exist" containerID="ce3177f7f1741daae4207fbb1f113367b20ff0348b314121a2c0a0cf3382651d" Sep 14 03:07:38.657873 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:38.657787 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce3177f7f1741daae4207fbb1f113367b20ff0348b314121a2c0a0cf3382651d"} err="failed to get container status \"ce3177f7f1741daae4207fbb1f113367b20ff0348b314121a2c0a0cf3382651d\": rpc error: code = NotFound desc = could not find container \"ce3177f7f1741daae4207fbb1f113367b20ff0348b314121a2c0a0cf3382651d\": container with ID starting with ce3177f7f1741daae4207fbb1f113367b20ff0348b314121a2c0a0cf3382651d not found: ID does not exist" Sep 14 03:07:38.657873 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:38.657806 2564 scope.go:117] "RemoveContainer" containerID="96296bd7709f22bd32d93f84b16a158b176bf139f4d2e43fdbddd6dc10d829c8" Sep 14 03:07:38.658079 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:07:38.658060 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96296bd7709f22bd32d93f84b16a158b176bf139f4d2e43fdbddd6dc10d829c8\": container with ID starting with 96296bd7709f22bd32d93f84b16a158b176bf139f4d2e43fdbddd6dc10d829c8 not found: ID does not exist" containerID="96296bd7709f22bd32d93f84b16a158b176bf139f4d2e43fdbddd6dc10d829c8" Sep 14 03:07:38.658126 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:38.658086 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96296bd7709f22bd32d93f84b16a158b176bf139f4d2e43fdbddd6dc10d829c8"} err="failed to get container status \"96296bd7709f22bd32d93f84b16a158b176bf139f4d2e43fdbddd6dc10d829c8\": rpc error: code = NotFound desc = could not find container \"96296bd7709f22bd32d93f84b16a158b176bf139f4d2e43fdbddd6dc10d829c8\": container with ID starting with 96296bd7709f22bd32d93f84b16a158b176bf139f4d2e43fdbddd6dc10d829c8 not found: ID does not exist" Sep 14 03:07:38.660855 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:38.660832 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8"] Sep 14 03:07:38.665338 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:38.665315 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-853daece-openshift-default-664896b9b-xkbg8"] Sep 14 03:07:38.751403 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:38.751361 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0146a9b6-c656-4cc7-ba7c-b49e54150941" path="/var/lib/kubelet/pods/0146a9b6-c656-4cc7-ba7c-b49e54150941/volumes" Sep 14 03:07:40.988102 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:40.988061 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:41.134456 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:41.134419 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:41.246249 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:41.246146 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:41.811343 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:41.811302 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:45.631373 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:45.631325 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:45.661227 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:45.661189 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4"] Sep 14 03:07:45.661523 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:45.661453 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4" podUID="38f2b9d3-868c-49d3-bfd9-24fd31e81e9d" containerName="payload-processing" containerID="cri-o://abb5ffbd28a930ed6673372a784cd9bb27f932c2bfc13fbda1d758392d719c36" gracePeriod=30 Sep 14 03:07:45.674385 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:45.674351 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47"] Sep 14 03:07:45.674795 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:45.674769 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47" podUID="86f58e93-1d50-418d-887e-a7fb31419cc3" containerName="payload-pre-processing" containerID="cri-o://8a5d918e42b7e1bced37a06956f2621a3d1e594e4f2f97dce3e9578148ac94fd" gracePeriod=30 Sep 14 03:07:45.706544 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:45.706510 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:45.733346 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:45.733316 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4" podUID="38f2b9d3-868c-49d3-bfd9-24fd31e81e9d" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.39:9004: connect: connection refused" Sep 14 03:07:45.733438 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:45.733316 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47" podUID="86f58e93-1d50-418d-887e-a7fb31419cc3" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.38:9004: connect: connection refused" Sep 14 03:07:45.788723 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:45.788672 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:45.880386 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:45.880093 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:45.958040 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:45.958016 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4" Sep 14 03:07:45.960862 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:45.960839 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47" Sep 14 03:07:46.129785 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.129733 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f58e93-1d50-418d-887e-a7fb31419cc3-plugins-config-volume\") pod \"86f58e93-1d50-418d-887e-a7fb31419cc3\" (UID: \"86f58e93-1d50-418d-887e-a7fb31419cc3\") " Sep 14 03:07:46.130038 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.129824 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4d6cr\" (UniqueName: \"kubernetes.io/projected/38f2b9d3-868c-49d3-bfd9-24fd31e81e9d-kube-api-access-4d6cr\") pod \"38f2b9d3-868c-49d3-bfd9-24fd31e81e9d\" (UID: \"38f2b9d3-868c-49d3-bfd9-24fd31e81e9d\") " Sep 14 03:07:46.130038 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.129876 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f2b9d3-868c-49d3-bfd9-24fd31e81e9d-plugins-config-volume\") pod \"38f2b9d3-868c-49d3-bfd9-24fd31e81e9d\" (UID: \"38f2b9d3-868c-49d3-bfd9-24fd31e81e9d\") " Sep 14 03:07:46.130038 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.129925 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkkqk\" (UniqueName: \"kubernetes.io/projected/86f58e93-1d50-418d-887e-a7fb31419cc3-kube-api-access-gkkqk\") pod \"86f58e93-1d50-418d-887e-a7fb31419cc3\" (UID: \"86f58e93-1d50-418d-887e-a7fb31419cc3\") " Sep 14 03:07:46.130272 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.130242 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86f58e93-1d50-418d-887e-a7fb31419cc3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "86f58e93-1d50-418d-887e-a7fb31419cc3" (UID: "86f58e93-1d50-418d-887e-a7fb31419cc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:07:46.130322 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.130293 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38f2b9d3-868c-49d3-bfd9-24fd31e81e9d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "38f2b9d3-868c-49d3-bfd9-24fd31e81e9d" (UID: "38f2b9d3-868c-49d3-bfd9-24fd31e81e9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:07:46.132212 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.132189 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f58e93-1d50-418d-887e-a7fb31419cc3-kube-api-access-gkkqk" (OuterVolumeSpecName: "kube-api-access-gkkqk") pod "86f58e93-1d50-418d-887e-a7fb31419cc3" (UID: "86f58e93-1d50-418d-887e-a7fb31419cc3"). InnerVolumeSpecName "kube-api-access-gkkqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:07:46.132302 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.132250 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f2b9d3-868c-49d3-bfd9-24fd31e81e9d-kube-api-access-4d6cr" (OuterVolumeSpecName: "kube-api-access-4d6cr") pod "38f2b9d3-868c-49d3-bfd9-24fd31e81e9d" (UID: "38f2b9d3-868c-49d3-bfd9-24fd31e81e9d"). InnerVolumeSpecName "kube-api-access-4d6cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:07:46.191279 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.191239 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:46.231203 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.231163 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f58e93-1d50-418d-887e-a7fb31419cc3-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:46.231203 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.231196 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d6cr\" (UniqueName: \"kubernetes.io/projected/38f2b9d3-868c-49d3-bfd9-24fd31e81e9d-kube-api-access-4d6cr\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:46.231203 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.231213 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38f2b9d3-868c-49d3-bfd9-24fd31e81e9d-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:46.231432 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.231224 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkkqk\" (UniqueName: \"kubernetes.io/projected/86f58e93-1d50-418d-887e-a7fb31419cc3-kube-api-access-gkkqk\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:07:46.292450 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.292404 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:46.682200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.682167 2564 generic.go:356] "Generic (PLEG): container finished" podID="38f2b9d3-868c-49d3-bfd9-24fd31e81e9d" containerID="abb5ffbd28a930ed6673372a784cd9bb27f932c2bfc13fbda1d758392d719c36" exitCode=0 Sep 14 03:07:46.682700 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.682226 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4" Sep 14 03:07:46.682700 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.682262 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4" event={"ID":"38f2b9d3-868c-49d3-bfd9-24fd31e81e9d","Type":"ContainerDied","Data":"abb5ffbd28a930ed6673372a784cd9bb27f932c2bfc13fbda1d758392d719c36"} Sep 14 03:07:46.682700 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.682325 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4" event={"ID":"38f2b9d3-868c-49d3-bfd9-24fd31e81e9d","Type":"ContainerDied","Data":"bf4c1a105e02318b55cce30018c7e8c6c4d89881deaf0d6cabd256f3704c37c2"} Sep 14 03:07:46.682700 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.682360 2564 scope.go:117] "RemoveContainer" containerID="abb5ffbd28a930ed6673372a784cd9bb27f932c2bfc13fbda1d758392d719c36" Sep 14 03:07:46.687647 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.687442 2564 generic.go:356] "Generic (PLEG): container finished" podID="86f58e93-1d50-418d-887e-a7fb31419cc3" containerID="8a5d918e42b7e1bced37a06956f2621a3d1e594e4f2f97dce3e9578148ac94fd" exitCode=0 Sep 14 03:07:46.687647 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.687577 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47" event={"ID":"86f58e93-1d50-418d-887e-a7fb31419cc3","Type":"ContainerDied","Data":"8a5d918e42b7e1bced37a06956f2621a3d1e594e4f2f97dce3e9578148ac94fd"} Sep 14 03:07:46.687647 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.687592 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47" Sep 14 03:07:46.687647 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.687623 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47" event={"ID":"86f58e93-1d50-418d-887e-a7fb31419cc3","Type":"ContainerDied","Data":"115f30385f9890603bd4f1f94c9b1d62885df161566f2541ab7da9936892c8be"} Sep 14 03:07:46.697378 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.697354 2564 scope.go:117] "RemoveContainer" containerID="abb5ffbd28a930ed6673372a784cd9bb27f932c2bfc13fbda1d758392d719c36" Sep 14 03:07:46.697798 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:07:46.697777 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb5ffbd28a930ed6673372a784cd9bb27f932c2bfc13fbda1d758392d719c36\": container with ID starting with abb5ffbd28a930ed6673372a784cd9bb27f932c2bfc13fbda1d758392d719c36 not found: ID does not exist" containerID="abb5ffbd28a930ed6673372a784cd9bb27f932c2bfc13fbda1d758392d719c36" Sep 14 03:07:46.697901 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.697816 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb5ffbd28a930ed6673372a784cd9bb27f932c2bfc13fbda1d758392d719c36"} err="failed to get container status \"abb5ffbd28a930ed6673372a784cd9bb27f932c2bfc13fbda1d758392d719c36\": rpc error: code = NotFound desc = could not find container \"abb5ffbd28a930ed6673372a784cd9bb27f932c2bfc13fbda1d758392d719c36\": container with ID starting with abb5ffbd28a930ed6673372a784cd9bb27f932c2bfc13fbda1d758392d719c36 not found: ID does not exist" Sep 14 03:07:46.697901 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.697841 2564 scope.go:117] "RemoveContainer" containerID="8a5d918e42b7e1bced37a06956f2621a3d1e594e4f2f97dce3e9578148ac94fd" Sep 14 03:07:46.706869 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.706841 2564 scope.go:117] "RemoveContainer" containerID="8a5d918e42b7e1bced37a06956f2621a3d1e594e4f2f97dce3e9578148ac94fd" Sep 14 03:07:46.707133 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:07:46.707113 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a5d918e42b7e1bced37a06956f2621a3d1e594e4f2f97dce3e9578148ac94fd\": container with ID starting with 8a5d918e42b7e1bced37a06956f2621a3d1e594e4f2f97dce3e9578148ac94fd not found: ID does not exist" containerID="8a5d918e42b7e1bced37a06956f2621a3d1e594e4f2f97dce3e9578148ac94fd" Sep 14 03:07:46.707193 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.707145 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a5d918e42b7e1bced37a06956f2621a3d1e594e4f2f97dce3e9578148ac94fd"} err="failed to get container status \"8a5d918e42b7e1bced37a06956f2621a3d1e594e4f2f97dce3e9578148ac94fd\": rpc error: code = NotFound desc = could not find container \"8a5d918e42b7e1bced37a06956f2621a3d1e594e4f2f97dce3e9578148ac94fd\": container with ID starting with 8a5d918e42b7e1bced37a06956f2621a3d1e594e4f2f97dce3e9578148ac94fd not found: ID does not exist" Sep 14 03:07:46.710568 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.710543 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4"] Sep 14 03:07:46.715966 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.715943 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bc1117-5ffb4f5889-z4lq4"] Sep 14 03:07:46.728074 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.728048 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47"] Sep 14 03:07:46.730727 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.730702 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bc1117-84fd687bc8-zbz47"] Sep 14 03:07:46.750679 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.750651 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38f2b9d3-868c-49d3-bfd9-24fd31e81e9d" path="/var/lib/kubelet/pods/38f2b9d3-868c-49d3-bfd9-24fd31e81e9d/volumes" Sep 14 03:07:46.751009 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:46.750997 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86f58e93-1d50-418d-887e-a7fb31419cc3" path="/var/lib/kubelet/pods/86f58e93-1d50-418d-887e-a7fb31419cc3/volumes" Sep 14 03:07:52.455512 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:52.455457 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:52.567019 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:52.566981 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:52.779373 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:52.779279 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:53.310148 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.310110 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:53.598751 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.598664 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz"] Sep 14 03:07:53.599082 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.599031 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38f2b9d3-868c-49d3-bfd9-24fd31e81e9d" containerName="payload-processing" Sep 14 03:07:53.599082 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.599042 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="38f2b9d3-868c-49d3-bfd9-24fd31e81e9d" containerName="payload-processing" Sep 14 03:07:53.599082 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.599054 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81882939-af3b-4659-909b-8463416737dc" containerName="payload-pre-processing" Sep 14 03:07:53.599082 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.599060 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="81882939-af3b-4659-909b-8463416737dc" containerName="payload-pre-processing" Sep 14 03:07:53.599082 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.599069 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31915719-a15a-449f-814f-84f35762a374" containerName="payload-processing" Sep 14 03:07:53.599082 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.599075 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="31915719-a15a-449f-814f-84f35762a374" containerName="payload-processing" Sep 14 03:07:53.599296 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.599084 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86f58e93-1d50-418d-887e-a7fb31419cc3" containerName="payload-pre-processing" Sep 14 03:07:53.599296 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.599091 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="86f58e93-1d50-418d-887e-a7fb31419cc3" containerName="payload-pre-processing" Sep 14 03:07:53.599296 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.599097 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0146a9b6-c656-4cc7-ba7c-b49e54150941" containerName="istio-proxy" Sep 14 03:07:53.599296 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.599102 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0146a9b6-c656-4cc7-ba7c-b49e54150941" containerName="istio-proxy" Sep 14 03:07:53.599296 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.599111 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd13925-79e3-47c8-95df-c5cbf08e1f14" containerName="payload-processing" Sep 14 03:07:53.599296 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.599117 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd13925-79e3-47c8-95df-c5cbf08e1f14" containerName="payload-processing" Sep 14 03:07:53.599296 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.599125 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0146a9b6-c656-4cc7-ba7c-b49e54150941" containerName="istio-proxy" Sep 14 03:07:53.599296 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.599130 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0146a9b6-c656-4cc7-ba7c-b49e54150941" containerName="istio-proxy" Sep 14 03:07:53.599296 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.599186 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd13925-79e3-47c8-95df-c5cbf08e1f14" containerName="payload-processing" Sep 14 03:07:53.599296 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.599198 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0146a9b6-c656-4cc7-ba7c-b49e54150941" containerName="istio-proxy" Sep 14 03:07:53.599296 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.599208 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0146a9b6-c656-4cc7-ba7c-b49e54150941" containerName="istio-proxy" Sep 14 03:07:53.599296 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.599214 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="38f2b9d3-868c-49d3-bfd9-24fd31e81e9d" containerName="payload-processing" Sep 14 03:07:53.599296 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.599223 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="86f58e93-1d50-418d-887e-a7fb31419cc3" containerName="payload-pre-processing" Sep 14 03:07:53.599296 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.599228 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="31915719-a15a-449f-814f-84f35762a374" containerName="payload-processing" Sep 14 03:07:53.599296 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.599233 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="81882939-af3b-4659-909b-8463416737dc" containerName="payload-pre-processing" Sep 14 03:07:53.601860 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.601841 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.604572 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.604550 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 14 03:07:53.604683 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.604580 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-0c919e-openshift-default-dockercfg-bnjmc\"" Sep 14 03:07:53.615723 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.615692 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz"] Sep 14 03:07:53.626831 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.626775 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:53.694382 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.694344 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-envoy\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.694592 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.694408 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk7c9\" (UniqueName: \"kubernetes.io/projected/2e8a7b82-b28d-49b1-b806-9a53e60391df-kube-api-access-vk7c9\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.694592 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.694446 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-data\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.694592 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.694471 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-workload-certs\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.694592 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.694520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-workload-socket\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.694592 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.694536 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-credential-socket\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.694592 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.694555 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-token\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.694812 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.694611 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-podinfo\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.694812 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.694637 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8a7b82-b28d-49b1-b806-9a53e60391df-istiod-ca-cert\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.795128 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.795094 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-envoy\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.795353 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.795164 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk7c9\" (UniqueName: \"kubernetes.io/projected/2e8a7b82-b28d-49b1-b806-9a53e60391df-kube-api-access-vk7c9\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.795353 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.795184 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-data\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.795353 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.795222 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-workload-certs\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.795353 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.795250 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-workload-socket\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.795353 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.795276 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-credential-socket\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.795353 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.795305 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-token\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.795353 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.795353 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-podinfo\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.795759 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.795376 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8a7b82-b28d-49b1-b806-9a53e60391df-istiod-ca-cert\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.795759 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.795653 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-data\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.795869 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.795800 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-credential-socket\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.795921 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.795876 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-workload-certs\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.795921 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.795887 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-workload-socket\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.796208 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.796183 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8a7b82-b28d-49b1-b806-9a53e60391df-istiod-ca-cert\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.797779 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.797740 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-envoy\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.798354 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.798332 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-podinfo\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.804719 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.804698 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk7c9\" (UniqueName: \"kubernetes.io/projected/2e8a7b82-b28d-49b1-b806-9a53e60391df-kube-api-access-vk7c9\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.806233 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.806207 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-token\") pod \"e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:53.912177 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:53.912136 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:54.196584 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:54.196547 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz"] Sep 14 03:07:54.198271 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:07:54.198239 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8a7b82_b28d_49b1_b806_9a53e60391df.slice/crio-8d0ffed89067a09c2d99efc204aa10b0696d8456805aaf1cb63f8cdf7e4fab05 WatchSource:0}: Error finding container 8d0ffed89067a09c2d99efc204aa10b0696d8456805aaf1cb63f8cdf7e4fab05: Status 404 returned error can't find the container with id 8d0ffed89067a09c2d99efc204aa10b0696d8456805aaf1cb63f8cdf7e4fab05 Sep 14 03:07:54.200519 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:54.200463 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:07:54.200614 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:54.200558 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:07:54.200614 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:54.200587 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:07:54.414300 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:54.414255 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:54.731381 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:54.731341 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" event={"ID":"2e8a7b82-b28d-49b1-b806-9a53e60391df","Type":"ContainerStarted","Data":"6463dedbbc3ac9ffe97b14e850697d4c8a616e59c08189d0e6662db1a07298c2"} Sep 14 03:07:54.731837 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:54.731393 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" event={"ID":"2e8a7b82-b28d-49b1-b806-9a53e60391df","Type":"ContainerStarted","Data":"8d0ffed89067a09c2d99efc204aa10b0696d8456805aaf1cb63f8cdf7e4fab05"} Sep 14 03:07:54.753151 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:54.753104 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" podStartSLOduration=1.753087222 podStartE2EDuration="1.753087222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:07:54.750663305 +0000 UTC m=+1046.480322447" watchObservedRunningTime="2026-09-14 03:07:54.753087222 +0000 UTC m=+1046.482746339" Sep 14 03:07:54.913090 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:54.913056 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:55.917719 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:55.917690 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:56.690347 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:56.690317 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:56.742733 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:56.742702 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:56.743804 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:56.743781 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:07:56.810314 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:56.810261 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:57.876790 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:57.876562 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:57.972549 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:57.972502 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:58.368365 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.368333 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:58.463918 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.463884 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb"] Sep 14 03:07:58.466291 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.466262 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:58.467665 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.467646 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb" Sep 14 03:07:58.470266 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.470242 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-0c919e\"" Sep 14 03:07:58.479406 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.479379 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb"] Sep 14 03:07:58.502922 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.502883 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh"] Sep 14 03:07:58.506319 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.506294 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh" Sep 14 03:07:58.508578 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.508556 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-0c919e-dockercfg-6f6gp\"" Sep 14 03:07:58.518576 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.518552 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh"] Sep 14 03:07:58.540433 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.540399 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rjfh\" (UniqueName: \"kubernetes.io/projected/3a39416c-586b-4890-9164-cea97b576dfe-kube-api-access-2rjfh\") pod \"payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb\" (UID: \"3a39416c-586b-4890-9164-cea97b576dfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb" Sep 14 03:07:58.540623 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.540443 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59rhd\" (UniqueName: \"kubernetes.io/projected/95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc-kube-api-access-59rhd\") pod \"payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh\" (UID: \"95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh" Sep 14 03:07:58.540623 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.540549 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a39416c-586b-4890-9164-cea97b576dfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb\" (UID: \"3a39416c-586b-4890-9164-cea97b576dfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb" Sep 14 03:07:58.540623 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.540574 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh\" (UID: \"95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh" Sep 14 03:07:58.544487 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.544442 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:07:58.641217 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.641123 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rjfh\" (UniqueName: \"kubernetes.io/projected/3a39416c-586b-4890-9164-cea97b576dfe-kube-api-access-2rjfh\") pod \"payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb\" (UID: \"3a39416c-586b-4890-9164-cea97b576dfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb" Sep 14 03:07:58.641217 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.641185 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59rhd\" (UniqueName: \"kubernetes.io/projected/95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc-kube-api-access-59rhd\") pod \"payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh\" (UID: \"95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh" Sep 14 03:07:58.641430 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.641284 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a39416c-586b-4890-9164-cea97b576dfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb\" (UID: \"3a39416c-586b-4890-9164-cea97b576dfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb" Sep 14 03:07:58.641430 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.641321 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh\" (UID: \"95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh" Sep 14 03:07:58.642038 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.642016 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a39416c-586b-4890-9164-cea97b576dfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb\" (UID: \"3a39416c-586b-4890-9164-cea97b576dfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb" Sep 14 03:07:58.642038 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.642032 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh\" (UID: \"95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh" Sep 14 03:07:58.649772 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.649747 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59rhd\" (UniqueName: \"kubernetes.io/projected/95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc-kube-api-access-59rhd\") pod \"payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh\" (UID: \"95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh" Sep 14 03:07:58.649941 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.649923 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rjfh\" (UniqueName: \"kubernetes.io/projected/3a39416c-586b-4890-9164-cea97b576dfe-kube-api-access-2rjfh\") pod \"payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb\" (UID: \"3a39416c-586b-4890-9164-cea97b576dfe\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb" Sep 14 03:07:58.779619 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.779587 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb" Sep 14 03:07:58.817593 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:58.817551 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh" Sep 14 03:07:59.055688 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:59.055658 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb"] Sep 14 03:07:59.058264 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:07:59.058234 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a39416c_586b_4890_9164_cea97b576dfe.slice/crio-82658ee2d50f3e6866343f221e393f74396207cf5f35f55d72870af801a9f2b2 WatchSource:0}: Error finding container 82658ee2d50f3e6866343f221e393f74396207cf5f35f55d72870af801a9f2b2: Status 404 returned error can't find the container with id 82658ee2d50f3e6866343f221e393f74396207cf5f35f55d72870af801a9f2b2 Sep 14 03:07:59.102493 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:59.102455 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh"] Sep 14 03:07:59.103988 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:07:59.103958 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c5dc92_cb1e_453d_b4f8_29cb8be0c4dc.slice/crio-8e85a9a936c6e919d1c75587e297a1850c46050d581e7116456e706011fed60d WatchSource:0}: Error finding container 8e85a9a936c6e919d1c75587e297a1850c46050d581e7116456e706011fed60d: Status 404 returned error can't find the container with id 8e85a9a936c6e919d1c75587e297a1850c46050d581e7116456e706011fed60d Sep 14 03:07:59.770539 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:59.770471 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh" event={"ID":"95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc","Type":"ContainerStarted","Data":"94ead2c0f92abb9c333874529fee8b02daec977cfdd8a5b9f98ff44f841e4b96"} Sep 14 03:07:59.770762 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:59.770566 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh" event={"ID":"95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc","Type":"ContainerStarted","Data":"8e85a9a936c6e919d1c75587e297a1850c46050d581e7116456e706011fed60d"} Sep 14 03:07:59.770762 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:59.770599 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh" Sep 14 03:07:59.772169 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:59.772140 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb" event={"ID":"3a39416c-586b-4890-9164-cea97b576dfe","Type":"ContainerStarted","Data":"d7412bc20f9cd591596c852c2953259e7272a7e0e947120c037166533520a9bb"} Sep 14 03:07:59.772270 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:59.772182 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb" event={"ID":"3a39416c-586b-4890-9164-cea97b576dfe","Type":"ContainerStarted","Data":"82658ee2d50f3e6866343f221e393f74396207cf5f35f55d72870af801a9f2b2"} Sep 14 03:07:59.772270 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:59.772201 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb" Sep 14 03:07:59.787204 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:59.787143 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh" podStartSLOduration=1.787124385 podStartE2EDuration="1.787124385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:07:59.7865645 +0000 UTC m=+1051.516223619" watchObservedRunningTime="2026-09-14 03:07:59.787124385 +0000 UTC m=+1051.516783525" Sep 14 03:07:59.804150 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:07:59.804084 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb" podStartSLOduration=1.804063943 podStartE2EDuration="1.804063943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:07:59.801906732 +0000 UTC m=+1051.531565875" watchObservedRunningTime="2026-09-14 03:07:59.804063943 +0000 UTC m=+1051.533723075" Sep 14 03:08:01.945577 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:01.945534 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:07.704526 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:07.704467 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:10.780077 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:10.780030 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh" Sep 14 03:08:10.780490 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:10.780126 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb" Sep 14 03:08:11.744659 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:11.744629 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 14 03:08:11.744830 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:11.744699 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" podUID="2e8a7b82-b28d-49b1-b806-9a53e60391df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 14 03:08:11.843752 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:11.843712 2564 generic.go:356] "Generic (PLEG): container finished" podID="2e8a7b82-b28d-49b1-b806-9a53e60391df" containerID="6463dedbbc3ac9ffe97b14e850697d4c8a616e59c08189d0e6662db1a07298c2" exitCode=0 Sep 14 03:08:11.844124 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:11.843789 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" event={"ID":"2e8a7b82-b28d-49b1-b806-9a53e60391df","Type":"ContainerDied","Data":"6463dedbbc3ac9ffe97b14e850697d4c8a616e59c08189d0e6662db1a07298c2"} Sep 14 03:08:11.844220 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:11.844206 2564 scope.go:117] "RemoveContainer" containerID="6463dedbbc3ac9ffe97b14e850697d4c8a616e59c08189d0e6662db1a07298c2" Sep 14 03:08:11.845266 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:11.845232 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:08:11.845370 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:11.845318 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:08:11.845432 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:11.845373 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:08:12.850563 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:12.850528 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" event={"ID":"2e8a7b82-b28d-49b1-b806-9a53e60391df","Type":"ContainerStarted","Data":"b9b1fd12b8a7be60f48da215872265594ed948c359d8e767a29ba6e67f4d3413"} Sep 14 03:08:12.912832 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:12.912796 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:08:12.913075 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:12.913054 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 14 03:08:12.913115 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:12.913102 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" podUID="2e8a7b82-b28d-49b1-b806-9a53e60391df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 14 03:08:13.912574 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:13.912539 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 14 03:08:13.912958 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:13.912603 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" podUID="2e8a7b82-b28d-49b1-b806-9a53e60391df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 14 03:08:14.258941 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:14.258842 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:14.331168 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:14.331132 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:14.444839 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:14.444801 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:14.554146 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:14.554049 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:14.913074 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:14.913040 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 14 03:08:14.913587 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:14.913095 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" podUID="2e8a7b82-b28d-49b1-b806-9a53e60391df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 14 03:08:15.912912 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:15.912867 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 14 03:08:15.913102 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:15.912936 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" podUID="2e8a7b82-b28d-49b1-b806-9a53e60391df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 14 03:08:16.913363 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:16.913323 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Sep 14 03:08:16.913798 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:16.913390 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" podUID="2e8a7b82-b28d-49b1-b806-9a53e60391df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Sep 14 03:08:17.916365 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:17.916334 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:08:17.916873 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:17.916801 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:08:17.917453 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:17.917436 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:08:19.741727 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:19.741641 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:20.420625 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:20.420579 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:21.735013 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:21.734974 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:22.129633 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:22.129545 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:22.422264 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:22.422224 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:22.704184 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:22.704094 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:24.584807 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:24.584769 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:25.104057 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:25.104023 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:25.229107 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:25.229073 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:25.329780 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:25.329736 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:26.530866 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:26.530831 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:26.593191 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:26.593153 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:26.661530 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:26.661491 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:27.181322 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.181287 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:27.334497 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.334454 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67"] Sep 14 03:08:27.338042 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.338016 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67" Sep 14 03:08:27.345763 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.345737 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-f83c87-dockercfg-2qlm9\"" Sep 14 03:08:27.346036 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.346018 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-f83c87\"" Sep 14 03:08:27.357458 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.357434 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67"] Sep 14 03:08:27.365741 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.365711 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:27.396320 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.396285 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7"] Sep 14 03:08:27.399789 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.399770 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7" Sep 14 03:08:27.411893 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.411872 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7"] Sep 14 03:08:27.426885 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.426860 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x297q\" (UniqueName: \"kubernetes.io/projected/347687cf-a121-436d-85d8-271fabc52911-kube-api-access-x297q\") pod \"payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67\" (UID: \"347687cf-a121-436d-85d8-271fabc52911\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67" Sep 14 03:08:27.427035 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.426899 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/347687cf-a121-436d-85d8-271fabc52911-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67\" (UID: \"347687cf-a121-436d-85d8-271fabc52911\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67" Sep 14 03:08:27.528150 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.528064 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae11342-3dfe-4a6f-8fb8-8d7ce5a78027-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7\" (UID: \"fae11342-3dfe-4a6f-8fb8-8d7ce5a78027\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7" Sep 14 03:08:27.528150 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.528106 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95zsw\" (UniqueName: \"kubernetes.io/projected/fae11342-3dfe-4a6f-8fb8-8d7ce5a78027-kube-api-access-95zsw\") pod \"payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7\" (UID: \"fae11342-3dfe-4a6f-8fb8-8d7ce5a78027\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7" Sep 14 03:08:27.528371 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.528190 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x297q\" (UniqueName: \"kubernetes.io/projected/347687cf-a121-436d-85d8-271fabc52911-kube-api-access-x297q\") pod \"payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67\" (UID: \"347687cf-a121-436d-85d8-271fabc52911\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67" Sep 14 03:08:27.528371 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.528229 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/347687cf-a121-436d-85d8-271fabc52911-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67\" (UID: \"347687cf-a121-436d-85d8-271fabc52911\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67" Sep 14 03:08:27.528910 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.528887 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/347687cf-a121-436d-85d8-271fabc52911-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67\" (UID: \"347687cf-a121-436d-85d8-271fabc52911\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67" Sep 14 03:08:27.536856 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.536834 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x297q\" (UniqueName: \"kubernetes.io/projected/347687cf-a121-436d-85d8-271fabc52911-kube-api-access-x297q\") pod \"payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67\" (UID: \"347687cf-a121-436d-85d8-271fabc52911\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67" Sep 14 03:08:27.629302 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.629266 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae11342-3dfe-4a6f-8fb8-8d7ce5a78027-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7\" (UID: \"fae11342-3dfe-4a6f-8fb8-8d7ce5a78027\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7" Sep 14 03:08:27.629302 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.629304 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95zsw\" (UniqueName: \"kubernetes.io/projected/fae11342-3dfe-4a6f-8fb8-8d7ce5a78027-kube-api-access-95zsw\") pod \"payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7\" (UID: \"fae11342-3dfe-4a6f-8fb8-8d7ce5a78027\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7" Sep 14 03:08:27.630069 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.630049 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae11342-3dfe-4a6f-8fb8-8d7ce5a78027-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7\" (UID: \"fae11342-3dfe-4a6f-8fb8-8d7ce5a78027\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7" Sep 14 03:08:27.637729 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.637694 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95zsw\" (UniqueName: \"kubernetes.io/projected/fae11342-3dfe-4a6f-8fb8-8d7ce5a78027-kube-api-access-95zsw\") pod \"payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7\" (UID: \"fae11342-3dfe-4a6f-8fb8-8d7ce5a78027\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7" Sep 14 03:08:27.648257 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.648237 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67" Sep 14 03:08:27.710264 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.710032 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7" Sep 14 03:08:27.925075 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.925042 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67"] Sep 14 03:08:27.927167 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:08:27.927144 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347687cf_a121_436d_85d8_271fabc52911.slice/crio-6a8a46307dab5b4b9cb8e4b167afa72e56021c8fb1d715786f32ac8049647a1e WatchSource:0}: Error finding container 6a8a46307dab5b4b9cb8e4b167afa72e56021c8fb1d715786f32ac8049647a1e: Status 404 returned error can't find the container with id 6a8a46307dab5b4b9cb8e4b167afa72e56021c8fb1d715786f32ac8049647a1e Sep 14 03:08:27.988563 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:27.988541 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7"] Sep 14 03:08:27.990877 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:08:27.990852 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae11342_3dfe_4a6f_8fb8_8d7ce5a78027.slice/crio-19f31062b111bc29904687716ca1cc9417569152232b2ed663769924309803a6 WatchSource:0}: Error finding container 19f31062b111bc29904687716ca1cc9417569152232b2ed663769924309803a6: Status 404 returned error can't find the container with id 19f31062b111bc29904687716ca1cc9417569152232b2ed663769924309803a6 Sep 14 03:08:28.942039 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:28.942004 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67" event={"ID":"347687cf-a121-436d-85d8-271fabc52911","Type":"ContainerStarted","Data":"893cdcb082081b6747f6a775f5304afb56827997ff4b821d5f54e1571591cb1b"} Sep 14 03:08:28.942499 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:28.942095 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67" event={"ID":"347687cf-a121-436d-85d8-271fabc52911","Type":"ContainerStarted","Data":"6a8a46307dab5b4b9cb8e4b167afa72e56021c8fb1d715786f32ac8049647a1e"} Sep 14 03:08:28.942499 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:28.942224 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67" Sep 14 03:08:28.943997 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:28.943972 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7" event={"ID":"fae11342-3dfe-4a6f-8fb8-8d7ce5a78027","Type":"ContainerStarted","Data":"5f62b7fdd454747c0e6e0fc88192dc7d2c7291babb00f76a84886729ed0d9e6e"} Sep 14 03:08:28.944091 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:28.944019 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7" event={"ID":"fae11342-3dfe-4a6f-8fb8-8d7ce5a78027","Type":"ContainerStarted","Data":"19f31062b111bc29904687716ca1cc9417569152232b2ed663769924309803a6"} Sep 14 03:08:28.944091 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:28.944069 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7" Sep 14 03:08:28.959848 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:28.959805 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67" podStartSLOduration=1.959792467 podStartE2EDuration="1.959792467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:08:28.958157559 +0000 UTC m=+1080.687816678" watchObservedRunningTime="2026-09-14 03:08:28.959792467 +0000 UTC m=+1080.689451584" Sep 14 03:08:28.977950 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:28.977893 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7" podStartSLOduration=1.97787475 podStartE2EDuration="1.97787475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:08:28.974833023 +0000 UTC m=+1080.704492142" watchObservedRunningTime="2026-09-14 03:08:28.97787475 +0000 UTC m=+1080.707533870" Sep 14 03:08:30.352331 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:30.352300 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:36.460869 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:36.460827 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:36.568880 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:36.568842 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:36.664190 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:36.664147 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:39.951494 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:39.951452 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7" Sep 14 03:08:39.951941 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:39.951533 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67" Sep 14 03:08:47.203373 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:47.203340 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:47.987732 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:47.987698 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:48.145835 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:48.145802 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:48.254415 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:48.254321 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:48.786037 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:48.786003 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:49.338774 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:49.338735 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:50.222007 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:50.221975 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:50.323321 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:50.323287 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:54.328944 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:54.328906 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:54.440862 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:54.440829 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:54.549086 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:54.549037 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:54.648874 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:54.648842 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:55.312358 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:55.312315 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:55.412327 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:55.412281 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:55.510313 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:55.510269 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:57.035669 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:57.035636 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:08:58.009620 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:08:58.009582 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:00.502449 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:00.502408 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:04.106803 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:04.106767 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:04.210716 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:04.210683 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:04.825190 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:04.825155 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:11.154295 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:11.154255 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:11.255195 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:11.255151 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:11.416844 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:11.416754 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:11.729577 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:11.729492 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:12.083182 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:12.083092 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:14.315122 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:14.315076 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:14.615313 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:14.615233 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:14.709974 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:14.709930 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:15.104703 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:15.104668 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:15.923891 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:15.923853 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:16.009758 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.009690 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:16.190937 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.190839 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:16.572052 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.571969 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:16.667953 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.667913 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:16.693140 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.693105 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22"] Sep 14 03:09:16.696397 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.696372 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22" Sep 14 03:09:16.698653 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.698635 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-dfbaeab8\"" Sep 14 03:09:16.709251 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.709225 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22"] Sep 14 03:09:16.742706 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.742672 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6"] Sep 14 03:09:16.750208 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.750165 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6" Sep 14 03:09:16.752781 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.752755 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-dfbaeab8-dockercfg-4chfb\"" Sep 14 03:09:16.758867 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.758835 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6"] Sep 14 03:09:16.802006 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.801972 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbwgs\" (UniqueName: \"kubernetes.io/projected/c85b9076-9c86-4cf1-8c45-9913a7fd3b7a-kube-api-access-cbwgs\") pod \"payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6\" (UID: \"c85b9076-9c86-4cf1-8c45-9913a7fd3b7a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6" Sep 14 03:09:16.802226 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.802032 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgkgj\" (UniqueName: \"kubernetes.io/projected/a77cd3b6-c39b-4936-b490-3bb161e100a6-kube-api-access-vgkgj\") pod \"payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22\" (UID: \"a77cd3b6-c39b-4936-b490-3bb161e100a6\") " pod="openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22" Sep 14 03:09:16.802226 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.802101 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a77cd3b6-c39b-4936-b490-3bb161e100a6-plugins-config-volume\") pod \"payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22\" (UID: \"a77cd3b6-c39b-4936-b490-3bb161e100a6\") " pod="openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22" Sep 14 03:09:16.802226 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.802147 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85b9076-9c86-4cf1-8c45-9913a7fd3b7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6\" (UID: \"c85b9076-9c86-4cf1-8c45-9913a7fd3b7a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6" Sep 14 03:09:16.902881 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.902848 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbwgs\" (UniqueName: \"kubernetes.io/projected/c85b9076-9c86-4cf1-8c45-9913a7fd3b7a-kube-api-access-cbwgs\") pod \"payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6\" (UID: \"c85b9076-9c86-4cf1-8c45-9913a7fd3b7a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6" Sep 14 03:09:16.903053 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.902904 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgkgj\" (UniqueName: \"kubernetes.io/projected/a77cd3b6-c39b-4936-b490-3bb161e100a6-kube-api-access-vgkgj\") pod \"payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22\" (UID: \"a77cd3b6-c39b-4936-b490-3bb161e100a6\") " pod="openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22" Sep 14 03:09:16.903053 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.902938 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a77cd3b6-c39b-4936-b490-3bb161e100a6-plugins-config-volume\") pod \"payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22\" (UID: \"a77cd3b6-c39b-4936-b490-3bb161e100a6\") " pod="openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22" Sep 14 03:09:16.903053 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.902978 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85b9076-9c86-4cf1-8c45-9913a7fd3b7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6\" (UID: \"c85b9076-9c86-4cf1-8c45-9913a7fd3b7a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6" Sep 14 03:09:16.903703 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.903684 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85b9076-9c86-4cf1-8c45-9913a7fd3b7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6\" (UID: \"c85b9076-9c86-4cf1-8c45-9913a7fd3b7a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6" Sep 14 03:09:16.903814 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.903798 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a77cd3b6-c39b-4936-b490-3bb161e100a6-plugins-config-volume\") pod \"payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22\" (UID: \"a77cd3b6-c39b-4936-b490-3bb161e100a6\") " pod="openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22" Sep 14 03:09:16.912820 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.912792 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbwgs\" (UniqueName: \"kubernetes.io/projected/c85b9076-9c86-4cf1-8c45-9913a7fd3b7a-kube-api-access-cbwgs\") pod \"payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6\" (UID: \"c85b9076-9c86-4cf1-8c45-9913a7fd3b7a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6" Sep 14 03:09:16.912979 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:16.912792 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgkgj\" (UniqueName: \"kubernetes.io/projected/a77cd3b6-c39b-4936-b490-3bb161e100a6-kube-api-access-vgkgj\") pod \"payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22\" (UID: \"a77cd3b6-c39b-4936-b490-3bb161e100a6\") " pod="openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22" Sep 14 03:09:17.007546 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:17.007467 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22" Sep 14 03:09:17.063911 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:17.063877 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6" Sep 14 03:09:17.289261 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:17.289230 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22"] Sep 14 03:09:17.290239 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:09:17.290209 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77cd3b6_c39b_4936_b490_3bb161e100a6.slice/crio-2434e4b973e4a10bf9ca67842be690607599015f580653d96567eb5677aa611a WatchSource:0}: Error finding container 2434e4b973e4a10bf9ca67842be690607599015f580653d96567eb5677aa611a: Status 404 returned error can't find the container with id 2434e4b973e4a10bf9ca67842be690607599015f580653d96567eb5677aa611a Sep 14 03:09:17.356364 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:17.356329 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6"] Sep 14 03:09:17.358720 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:09:17.358699 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85b9076_9c86_4cf1_8c45_9913a7fd3b7a.slice/crio-fbfe054f77c47d21c723863b097d840f2eec99cf676d81e935b93ad69cea4544 WatchSource:0}: Error finding container fbfe054f77c47d21c723863b097d840f2eec99cf676d81e935b93ad69cea4544: Status 404 returned error can't find the container with id fbfe054f77c47d21c723863b097d840f2eec99cf676d81e935b93ad69cea4544 Sep 14 03:09:18.192858 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:18.192820 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22" event={"ID":"a77cd3b6-c39b-4936-b490-3bb161e100a6","Type":"ContainerStarted","Data":"43d7e6d89a197c08989b30fc96d9655e1cdfd8a7031fedb5c7ab2ea2a236b5d8"} Sep 14 03:09:18.193430 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:18.193403 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22" event={"ID":"a77cd3b6-c39b-4936-b490-3bb161e100a6","Type":"ContainerStarted","Data":"2434e4b973e4a10bf9ca67842be690607599015f580653d96567eb5677aa611a"} Sep 14 03:09:18.193582 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:18.193446 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22" Sep 14 03:09:18.194548 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:18.194524 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6" event={"ID":"c85b9076-9c86-4cf1-8c45-9913a7fd3b7a","Type":"ContainerStarted","Data":"8e98f8e235f78dc63418eb1a9a287646db0f0d2d2505f688266d6fe70f3e693a"} Sep 14 03:09:18.194664 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:18.194569 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6" event={"ID":"c85b9076-9c86-4cf1-8c45-9913a7fd3b7a","Type":"ContainerStarted","Data":"fbfe054f77c47d21c723863b097d840f2eec99cf676d81e935b93ad69cea4544"} Sep 14 03:09:18.194664 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:18.194654 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6" Sep 14 03:09:18.214401 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:18.214337 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22" podStartSLOduration=2.214319705 podStartE2EDuration="2.214319705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:09:18.211958873 +0000 UTC m=+1129.941618015" watchObservedRunningTime="2026-09-14 03:09:18.214319705 +0000 UTC m=+1129.943978822" Sep 14 03:09:18.232623 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:18.232558 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6" podStartSLOduration=2.232537466 podStartE2EDuration="2.232537466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:09:18.231521333 +0000 UTC m=+1129.961180450" watchObservedRunningTime="2026-09-14 03:09:18.232537466 +0000 UTC m=+1129.962196604" Sep 14 03:09:29.202501 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:29.202457 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6" Sep 14 03:09:29.202903 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:29.202672 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22" Sep 14 03:09:39.507726 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:39.507687 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:39.615188 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:39.615161 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:40.285234 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:40.285187 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:40.383182 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:40.383145 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:40.726164 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:40.726101 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:41.138309 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:41.138221 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:41.699616 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:41.699579 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:42.121822 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:42.121745 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:46.891635 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:46.891586 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:46.990002 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:46.989961 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:48.018780 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:48.018749 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:48.910382 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:48.910352 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:49.620591 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:49.620557 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:50.176934 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:50.176899 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:57.924566 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:57.924527 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:57.957707 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:57.957675 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22"] Sep 14 03:09:57.957970 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:57.957942 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22" podUID="a77cd3b6-c39b-4936-b490-3bb161e100a6" containerName="payload-processing" containerID="cri-o://43d7e6d89a197c08989b30fc96d9655e1cdfd8a7031fedb5c7ab2ea2a236b5d8" gracePeriod=30 Sep 14 03:09:57.963758 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:57.963727 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6"] Sep 14 03:09:57.963989 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:57.963964 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6" podUID="c85b9076-9c86-4cf1-8c45-9913a7fd3b7a" containerName="payload-pre-processing" containerID="cri-o://8e98f8e235f78dc63418eb1a9a287646db0f0d2d2505f688266d6fe70f3e693a" gracePeriod=30 Sep 14 03:09:58.001864 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.001836 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:58.092408 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.092368 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:58.182635 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.182436 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:58.230857 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.230837 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22" Sep 14 03:09:58.233724 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.233706 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6" Sep 14 03:09:58.281059 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.281029 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgkgj\" (UniqueName: \"kubernetes.io/projected/a77cd3b6-c39b-4936-b490-3bb161e100a6-kube-api-access-vgkgj\") pod \"a77cd3b6-c39b-4936-b490-3bb161e100a6\" (UID: \"a77cd3b6-c39b-4936-b490-3bb161e100a6\") " Sep 14 03:09:58.281206 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.281073 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbwgs\" (UniqueName: \"kubernetes.io/projected/c85b9076-9c86-4cf1-8c45-9913a7fd3b7a-kube-api-access-cbwgs\") pod \"c85b9076-9c86-4cf1-8c45-9913a7fd3b7a\" (UID: \"c85b9076-9c86-4cf1-8c45-9913a7fd3b7a\") " Sep 14 03:09:58.281206 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.281119 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a77cd3b6-c39b-4936-b490-3bb161e100a6-plugins-config-volume\") pod \"a77cd3b6-c39b-4936-b490-3bb161e100a6\" (UID: \"a77cd3b6-c39b-4936-b490-3bb161e100a6\") " Sep 14 03:09:58.281206 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.281166 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85b9076-9c86-4cf1-8c45-9913a7fd3b7a-plugins-config-volume\") pod \"c85b9076-9c86-4cf1-8c45-9913a7fd3b7a\" (UID: \"c85b9076-9c86-4cf1-8c45-9913a7fd3b7a\") " Sep 14 03:09:58.281584 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.281559 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a77cd3b6-c39b-4936-b490-3bb161e100a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a77cd3b6-c39b-4936-b490-3bb161e100a6" (UID: "a77cd3b6-c39b-4936-b490-3bb161e100a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:09:58.281691 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.281598 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85b9076-9c86-4cf1-8c45-9913a7fd3b7a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c85b9076-9c86-4cf1-8c45-9913a7fd3b7a" (UID: "c85b9076-9c86-4cf1-8c45-9913a7fd3b7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:09:58.283814 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.283793 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77cd3b6-c39b-4936-b490-3bb161e100a6-kube-api-access-vgkgj" (OuterVolumeSpecName: "kube-api-access-vgkgj") pod "a77cd3b6-c39b-4936-b490-3bb161e100a6" (UID: "a77cd3b6-c39b-4936-b490-3bb161e100a6"). InnerVolumeSpecName "kube-api-access-vgkgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:09:58.283814 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.283801 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85b9076-9c86-4cf1-8c45-9913a7fd3b7a-kube-api-access-cbwgs" (OuterVolumeSpecName: "kube-api-access-cbwgs") pod "c85b9076-9c86-4cf1-8c45-9913a7fd3b7a" (UID: "c85b9076-9c86-4cf1-8c45-9913a7fd3b7a"). InnerVolumeSpecName "kube-api-access-cbwgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:09:58.382356 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.382330 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgkgj\" (UniqueName: \"kubernetes.io/projected/a77cd3b6-c39b-4936-b490-3bb161e100a6-kube-api-access-vgkgj\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:09:58.382356 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.382353 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbwgs\" (UniqueName: \"kubernetes.io/projected/c85b9076-9c86-4cf1-8c45-9913a7fd3b7a-kube-api-access-cbwgs\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:09:58.382562 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.382362 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a77cd3b6-c39b-4936-b490-3bb161e100a6-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:09:58.382562 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.382373 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85b9076-9c86-4cf1-8c45-9913a7fd3b7a-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:09:58.402418 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.402394 2564 generic.go:356] "Generic (PLEG): container finished" podID="a77cd3b6-c39b-4936-b490-3bb161e100a6" containerID="43d7e6d89a197c08989b30fc96d9655e1cdfd8a7031fedb5c7ab2ea2a236b5d8" exitCode=0 Sep 14 03:09:58.402548 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.402441 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22" Sep 14 03:09:58.402548 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.402496 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22" event={"ID":"a77cd3b6-c39b-4936-b490-3bb161e100a6","Type":"ContainerDied","Data":"43d7e6d89a197c08989b30fc96d9655e1cdfd8a7031fedb5c7ab2ea2a236b5d8"} Sep 14 03:09:58.402650 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.402553 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22" event={"ID":"a77cd3b6-c39b-4936-b490-3bb161e100a6","Type":"ContainerDied","Data":"2434e4b973e4a10bf9ca67842be690607599015f580653d96567eb5677aa611a"} Sep 14 03:09:58.402650 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.402585 2564 scope.go:117] "RemoveContainer" containerID="43d7e6d89a197c08989b30fc96d9655e1cdfd8a7031fedb5c7ab2ea2a236b5d8" Sep 14 03:09:58.403882 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.403858 2564 generic.go:356] "Generic (PLEG): container finished" podID="c85b9076-9c86-4cf1-8c45-9913a7fd3b7a" containerID="8e98f8e235f78dc63418eb1a9a287646db0f0d2d2505f688266d6fe70f3e693a" exitCode=0 Sep 14 03:09:58.403991 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.403899 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6" event={"ID":"c85b9076-9c86-4cf1-8c45-9913a7fd3b7a","Type":"ContainerDied","Data":"8e98f8e235f78dc63418eb1a9a287646db0f0d2d2505f688266d6fe70f3e693a"} Sep 14 03:09:58.403991 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.403911 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6" Sep 14 03:09:58.403991 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.403942 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6" event={"ID":"c85b9076-9c86-4cf1-8c45-9913a7fd3b7a","Type":"ContainerDied","Data":"fbfe054f77c47d21c723863b097d840f2eec99cf676d81e935b93ad69cea4544"} Sep 14 03:09:58.412523 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.412506 2564 scope.go:117] "RemoveContainer" containerID="43d7e6d89a197c08989b30fc96d9655e1cdfd8a7031fedb5c7ab2ea2a236b5d8" Sep 14 03:09:58.412799 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:09:58.412781 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d7e6d89a197c08989b30fc96d9655e1cdfd8a7031fedb5c7ab2ea2a236b5d8\": container with ID starting with 43d7e6d89a197c08989b30fc96d9655e1cdfd8a7031fedb5c7ab2ea2a236b5d8 not found: ID does not exist" containerID="43d7e6d89a197c08989b30fc96d9655e1cdfd8a7031fedb5c7ab2ea2a236b5d8" Sep 14 03:09:58.412863 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.412804 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d7e6d89a197c08989b30fc96d9655e1cdfd8a7031fedb5c7ab2ea2a236b5d8"} err="failed to get container status \"43d7e6d89a197c08989b30fc96d9655e1cdfd8a7031fedb5c7ab2ea2a236b5d8\": rpc error: code = NotFound desc = could not find container \"43d7e6d89a197c08989b30fc96d9655e1cdfd8a7031fedb5c7ab2ea2a236b5d8\": container with ID starting with 43d7e6d89a197c08989b30fc96d9655e1cdfd8a7031fedb5c7ab2ea2a236b5d8 not found: ID does not exist" Sep 14 03:09:58.412863 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.412824 2564 scope.go:117] "RemoveContainer" containerID="8e98f8e235f78dc63418eb1a9a287646db0f0d2d2505f688266d6fe70f3e693a" Sep 14 03:09:58.420684 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.420666 2564 scope.go:117] "RemoveContainer" containerID="8e98f8e235f78dc63418eb1a9a287646db0f0d2d2505f688266d6fe70f3e693a" Sep 14 03:09:58.420909 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:09:58.420894 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e98f8e235f78dc63418eb1a9a287646db0f0d2d2505f688266d6fe70f3e693a\": container with ID starting with 8e98f8e235f78dc63418eb1a9a287646db0f0d2d2505f688266d6fe70f3e693a not found: ID does not exist" containerID="8e98f8e235f78dc63418eb1a9a287646db0f0d2d2505f688266d6fe70f3e693a" Sep 14 03:09:58.420961 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.420913 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e98f8e235f78dc63418eb1a9a287646db0f0d2d2505f688266d6fe70f3e693a"} err="failed to get container status \"8e98f8e235f78dc63418eb1a9a287646db0f0d2d2505f688266d6fe70f3e693a\": rpc error: code = NotFound desc = could not find container \"8e98f8e235f78dc63418eb1a9a287646db0f0d2d2505f688266d6fe70f3e693a\": container with ID starting with 8e98f8e235f78dc63418eb1a9a287646db0f0d2d2505f688266d6fe70f3e693a not found: ID does not exist" Sep 14 03:09:58.427139 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.427116 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22"] Sep 14 03:09:58.429668 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.429649 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-dfbaeab8-5c4c7998f7-2pj22"] Sep 14 03:09:58.440365 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.440315 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6"] Sep 14 03:09:58.448796 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.448774 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-dfbaeab8-74fdf89f66-bcld6"] Sep 14 03:09:58.513039 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.513015 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:58.613388 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.613364 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:58.755800 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.751176 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a77cd3b6-c39b-4936-b490-3bb161e100a6" path="/var/lib/kubelet/pods/a77cd3b6-c39b-4936-b490-3bb161e100a6/volumes" Sep 14 03:09:58.755800 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:58.751786 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c85b9076-9c86-4cf1-8c45-9913a7fd3b7a" path="/var/lib/kubelet/pods/c85b9076-9c86-4cf1-8c45-9913a7fd3b7a/volumes" Sep 14 03:09:59.092798 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:59.092725 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:09:59.810095 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:09:59.810059 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:05.066954 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:05.066910 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:05.167825 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:05.167785 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:05.341341 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:05.341237 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:05.744016 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:05.743971 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:05.833010 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:05.832970 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:05.957453 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:05.957421 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:12.537239 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:12.537199 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:12.693327 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:12.693292 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:12.940079 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:12.940047 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:13.046567 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:13.046534 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:19.095916 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:19.095886 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:19.197772 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:19.197725 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:20.043914 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:20.043877 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:20.379199 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:20.379100 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:20.606514 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:20.606458 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:23.332916 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:23.332882 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:23.911662 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:23.911624 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:24.008206 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:24.008169 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:24.207132 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:24.207026 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:24.897817 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:24.897781 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:25.879696 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:25.879654 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:25.980684 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:25.980645 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:26.359896 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.359861 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:26.457155 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.454071 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m"] Sep 14 03:10:26.457155 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.455056 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c85b9076-9c86-4cf1-8c45-9913a7fd3b7a" containerName="payload-pre-processing" Sep 14 03:10:26.457155 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.455079 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c85b9076-9c86-4cf1-8c45-9913a7fd3b7a" containerName="payload-pre-processing" Sep 14 03:10:26.457155 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.455110 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a77cd3b6-c39b-4936-b490-3bb161e100a6" containerName="payload-processing" Sep 14 03:10:26.457155 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.455120 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a77cd3b6-c39b-4936-b490-3bb161e100a6" containerName="payload-processing" Sep 14 03:10:26.457155 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.455316 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c85b9076-9c86-4cf1-8c45-9913a7fd3b7a" containerName="payload-pre-processing" Sep 14 03:10:26.457155 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.455337 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a77cd3b6-c39b-4936-b490-3bb161e100a6" containerName="payload-processing" Sep 14 03:10:26.459668 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.459510 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m" Sep 14 03:10:26.461447 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.461422 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:26.463016 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.462998 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-6e87d0\"" Sep 14 03:10:26.468768 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.468736 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m"] Sep 14 03:10:26.494504 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.494462 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm"] Sep 14 03:10:26.497682 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.497669 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm" Sep 14 03:10:26.499808 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.499792 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-6e87d0-dockercfg-cqdmk\"" Sep 14 03:10:26.509268 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.509244 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm"] Sep 14 03:10:26.528097 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.528073 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2j2s\" (UniqueName: \"kubernetes.io/projected/01ddce15-f22b-4942-a53b-b30b415315ff-kube-api-access-j2j2s\") pod \"payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m\" (UID: \"01ddce15-f22b-4942-a53b-b30b415315ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m" Sep 14 03:10:26.528197 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.528132 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01ddce15-f22b-4942-a53b-b30b415315ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m\" (UID: \"01ddce15-f22b-4942-a53b-b30b415315ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m" Sep 14 03:10:26.550711 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.550684 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:26.628876 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.628808 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01ddce15-f22b-4942-a53b-b30b415315ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m\" (UID: \"01ddce15-f22b-4942-a53b-b30b415315ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m" Sep 14 03:10:26.628876 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.628864 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfn7j\" (UniqueName: \"kubernetes.io/projected/6125ac4f-31f5-4a0c-a000-5ef315c3f11c-kube-api-access-sfn7j\") pod \"payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm\" (UID: \"6125ac4f-31f5-4a0c-a000-5ef315c3f11c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm" Sep 14 03:10:26.629028 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.628941 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2j2s\" (UniqueName: \"kubernetes.io/projected/01ddce15-f22b-4942-a53b-b30b415315ff-kube-api-access-j2j2s\") pod \"payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m\" (UID: \"01ddce15-f22b-4942-a53b-b30b415315ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m" Sep 14 03:10:26.629028 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.628971 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6125ac4f-31f5-4a0c-a000-5ef315c3f11c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm\" (UID: \"6125ac4f-31f5-4a0c-a000-5ef315c3f11c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm" Sep 14 03:10:26.629604 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.629581 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01ddce15-f22b-4942-a53b-b30b415315ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m\" (UID: \"01ddce15-f22b-4942-a53b-b30b415315ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m" Sep 14 03:10:26.638927 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.638903 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2j2s\" (UniqueName: \"kubernetes.io/projected/01ddce15-f22b-4942-a53b-b30b415315ff-kube-api-access-j2j2s\") pod \"payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m\" (UID: \"01ddce15-f22b-4942-a53b-b30b415315ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m" Sep 14 03:10:26.730229 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.730202 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6125ac4f-31f5-4a0c-a000-5ef315c3f11c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm\" (UID: \"6125ac4f-31f5-4a0c-a000-5ef315c3f11c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm" Sep 14 03:10:26.730383 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.730277 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfn7j\" (UniqueName: \"kubernetes.io/projected/6125ac4f-31f5-4a0c-a000-5ef315c3f11c-kube-api-access-sfn7j\") pod \"payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm\" (UID: \"6125ac4f-31f5-4a0c-a000-5ef315c3f11c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm" Sep 14 03:10:26.730847 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.730829 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6125ac4f-31f5-4a0c-a000-5ef315c3f11c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm\" (UID: \"6125ac4f-31f5-4a0c-a000-5ef315c3f11c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm" Sep 14 03:10:26.738233 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.738210 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfn7j\" (UniqueName: \"kubernetes.io/projected/6125ac4f-31f5-4a0c-a000-5ef315c3f11c-kube-api-access-sfn7j\") pod \"payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm\" (UID: \"6125ac4f-31f5-4a0c-a000-5ef315c3f11c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm" Sep 14 03:10:26.772472 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.772454 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m" Sep 14 03:10:26.808227 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:26.808188 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm" Sep 14 03:10:27.046665 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:27.046639 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m"] Sep 14 03:10:27.048490 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:10:27.048452 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ddce15_f22b_4942_a53b_b30b415315ff.slice/crio-5a621dede5ebf3b446af9bc73c2da8b12ddb2a74fd8571bf7b97a9b98463d2a9 WatchSource:0}: Error finding container 5a621dede5ebf3b446af9bc73c2da8b12ddb2a74fd8571bf7b97a9b98463d2a9: Status 404 returned error can't find the container with id 5a621dede5ebf3b446af9bc73c2da8b12ddb2a74fd8571bf7b97a9b98463d2a9 Sep 14 03:10:27.089354 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:27.089328 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm"] Sep 14 03:10:27.090944 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:10:27.090920 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6125ac4f_31f5_4a0c_a000_5ef315c3f11c.slice/crio-9f071e7b36d5a0da6a08497c0dd56fd0f6b935e79c4ac3041851d299660093b1 WatchSource:0}: Error finding container 9f071e7b36d5a0da6a08497c0dd56fd0f6b935e79c4ac3041851d299660093b1: Status 404 returned error can't find the container with id 9f071e7b36d5a0da6a08497c0dd56fd0f6b935e79c4ac3041851d299660093b1 Sep 14 03:10:27.553795 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:27.553757 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm" event={"ID":"6125ac4f-31f5-4a0c-a000-5ef315c3f11c","Type":"ContainerStarted","Data":"8b037616e0e554a6f44c5f136a69cb7a5f9ccc871d4c5f5d0c4dd9d8628cd0f7"} Sep 14 03:10:27.554017 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:27.553823 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm" event={"ID":"6125ac4f-31f5-4a0c-a000-5ef315c3f11c","Type":"ContainerStarted","Data":"9f071e7b36d5a0da6a08497c0dd56fd0f6b935e79c4ac3041851d299660093b1"} Sep 14 03:10:27.554017 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:27.553857 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm" Sep 14 03:10:27.555411 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:27.555383 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m" event={"ID":"01ddce15-f22b-4942-a53b-b30b415315ff","Type":"ContainerStarted","Data":"4250d55f220c5ac23658c9fd3758f58b20ccec37cbda905768dc62791cfb0462"} Sep 14 03:10:27.555573 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:27.555424 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m" event={"ID":"01ddce15-f22b-4942-a53b-b30b415315ff","Type":"ContainerStarted","Data":"5a621dede5ebf3b446af9bc73c2da8b12ddb2a74fd8571bf7b97a9b98463d2a9"} Sep 14 03:10:27.555573 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:27.555509 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m" Sep 14 03:10:27.572050 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:27.572013 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm" podStartSLOduration=1.572000244 podStartE2EDuration="1.572000244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:10:27.570945575 +0000 UTC m=+1199.300604715" watchObservedRunningTime="2026-09-14 03:10:27.572000244 +0000 UTC m=+1199.301659361" Sep 14 03:10:27.610135 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:27.610090 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m" podStartSLOduration=1.610073174 podStartE2EDuration="1.610073174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:10:27.607951202 +0000 UTC m=+1199.337610324" watchObservedRunningTime="2026-09-14 03:10:27.610073174 +0000 UTC m=+1199.339732294" Sep 14 03:10:28.815983 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:28.815953 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bwv6_4369115a-4032-4f5a-a3e4-c02efe665617/ovn-acl-logging/0.log" Sep 14 03:10:28.817601 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:28.817578 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bwv6_4369115a-4032-4f5a-a3e4-c02efe665617/ovn-acl-logging/0.log" Sep 14 03:10:38.563357 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:38.563328 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m" Sep 14 03:10:38.563781 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:38.563399 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm" Sep 14 03:10:47.916656 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:47.916617 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:48.785369 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:48.785333 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:48.898065 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:48.898033 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:49.246670 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:49.246631 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:49.514793 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:49.514671 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:49.609561 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:49.609527 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:49.810556 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:49.810465 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:50.240608 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:50.240572 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:52.329618 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:52.329585 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:53.054559 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:53.054527 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:53.151519 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:53.151459 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:53.253132 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:53.253101 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:54.228423 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:54.228385 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:54.934974 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:54.934940 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:55.624256 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:55.624223 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:55.726912 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:55.726879 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:55.823427 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:55.823397 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:56.095325 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:56.095266 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:56.178567 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:56.178535 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:10:56.203513 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:56.203467 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n"] Sep 14 03:10:56.207031 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:56.207016 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n" Sep 14 03:10:56.209391 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:56.209369 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-9f77a9\"" Sep 14 03:10:56.219107 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:56.219086 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n"] Sep 14 03:10:56.286838 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:56.286809 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202a2300-292b-4bb2-87e0-ce7dfe7526e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n\" (UID: \"202a2300-292b-4bb2-87e0-ce7dfe7526e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n" Sep 14 03:10:56.286957 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:56.286869 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pv5v\" (UniqueName: \"kubernetes.io/projected/202a2300-292b-4bb2-87e0-ce7dfe7526e4-kube-api-access-5pv5v\") pod \"payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n\" (UID: \"202a2300-292b-4bb2-87e0-ce7dfe7526e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n" Sep 14 03:10:56.387776 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:56.387709 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pv5v\" (UniqueName: \"kubernetes.io/projected/202a2300-292b-4bb2-87e0-ce7dfe7526e4-kube-api-access-5pv5v\") pod \"payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n\" (UID: \"202a2300-292b-4bb2-87e0-ce7dfe7526e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n" Sep 14 03:10:56.387890 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:56.387802 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202a2300-292b-4bb2-87e0-ce7dfe7526e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n\" (UID: \"202a2300-292b-4bb2-87e0-ce7dfe7526e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n" Sep 14 03:10:56.388444 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:56.388426 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202a2300-292b-4bb2-87e0-ce7dfe7526e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n\" (UID: \"202a2300-292b-4bb2-87e0-ce7dfe7526e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n" Sep 14 03:10:56.397635 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:56.397612 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pv5v\" (UniqueName: \"kubernetes.io/projected/202a2300-292b-4bb2-87e0-ce7dfe7526e4-kube-api-access-5pv5v\") pod \"payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n\" (UID: \"202a2300-292b-4bb2-87e0-ce7dfe7526e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n" Sep 14 03:10:56.518078 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:56.518051 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n" Sep 14 03:10:56.799711 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:56.799681 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n"] Sep 14 03:10:56.802834 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:10:56.802809 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202a2300_292b_4bb2_87e0_ce7dfe7526e4.slice/crio-09f9dc93f5d3290c3896b4b4a7d6c3433b04b59ca71b18cf213d5d497b0212ec WatchSource:0}: Error finding container 09f9dc93f5d3290c3896b4b4a7d6c3433b04b59ca71b18cf213d5d497b0212ec: Status 404 returned error can't find the container with id 09f9dc93f5d3290c3896b4b4a7d6c3433b04b59ca71b18cf213d5d497b0212ec Sep 14 03:10:57.718090 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:57.718060 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n" event={"ID":"202a2300-292b-4bb2-87e0-ce7dfe7526e4","Type":"ContainerStarted","Data":"1581f5e4af418e878ac05e349248678cda73cc5edd922e5a0028f82233995709"} Sep 14 03:10:57.718281 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:57.718108 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n" event={"ID":"202a2300-292b-4bb2-87e0-ce7dfe7526e4","Type":"ContainerStarted","Data":"09f9dc93f5d3290c3896b4b4a7d6c3433b04b59ca71b18cf213d5d497b0212ec"} Sep 14 03:10:57.718281 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:57.718176 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n" Sep 14 03:10:57.738858 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:10:57.738813 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n" podStartSLOduration=1.738798098 podStartE2EDuration="1.738798098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:10:57.736864235 +0000 UTC m=+1229.466523376" watchObservedRunningTime="2026-09-14 03:10:57.738798098 +0000 UTC m=+1229.468457214" Sep 14 03:11:08.724663 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:08.724632 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n" Sep 14 03:11:17.914598 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:17.914560 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:18.015850 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:18.015812 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:18.675819 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:18.675773 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:19.445625 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:19.445590 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:20.149254 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:20.149199 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:20.718523 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:20.718489 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:21.510863 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:21.510824 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:21.592395 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:21.592357 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:27.452051 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:27.452010 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:27.799822 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:27.799712 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:27.952441 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:27.952403 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:29.812246 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:29.812213 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:30.619511 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:30.619458 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:31.380872 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:31.380830 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:31.494215 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:31.494164 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:31.609979 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:31.609944 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:32.627306 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:32.627268 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:33.603675 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:33.603640 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:33.809969 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:33.809941 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:34.399210 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:34.399177 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:34.608276 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:34.608240 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:35.222976 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.222936 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:35.299228 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.299189 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:35.399017 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.398981 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:35.717463 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.717434 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:35.804780 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.804748 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:35.812160 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.812133 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9"] Sep 14 03:11:35.815898 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.815882 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9" Sep 14 03:11:35.818687 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.818665 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d6ac6dc8\"" Sep 14 03:11:35.818792 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.818713 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d6ac6dc8-dockercfg-l69qx\"" Sep 14 03:11:35.824147 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.824110 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9"] Sep 14 03:11:35.865573 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.865543 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln"] Sep 14 03:11:35.869270 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.869249 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln" Sep 14 03:11:35.877274 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.877247 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r4vd\" (UniqueName: \"kubernetes.io/projected/9c520de8-a92f-4869-bfd6-4d9cd1806f8a-kube-api-access-8r4vd\") pod \"payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9\" (UID: \"9c520de8-a92f-4869-bfd6-4d9cd1806f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9" Sep 14 03:11:35.877393 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.877370 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd32c34-3a3c-46bc-a639-7a9c0916a56f-plugins-config-volume\") pod \"payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln\" (UID: \"abd32c34-3a3c-46bc-a639-7a9c0916a56f\") " pod="openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln" Sep 14 03:11:35.877464 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.877416 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv9np\" (UniqueName: \"kubernetes.io/projected/abd32c34-3a3c-46bc-a639-7a9c0916a56f-kube-api-access-lv9np\") pod \"payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln\" (UID: \"abd32c34-3a3c-46bc-a639-7a9c0916a56f\") " pod="openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln" Sep 14 03:11:35.877639 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.877600 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c520de8-a92f-4869-bfd6-4d9cd1806f8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9\" (UID: \"9c520de8-a92f-4869-bfd6-4d9cd1806f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9" Sep 14 03:11:35.879430 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.879404 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln"] Sep 14 03:11:35.978651 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.978572 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c520de8-a92f-4869-bfd6-4d9cd1806f8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9\" (UID: \"9c520de8-a92f-4869-bfd6-4d9cd1806f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9" Sep 14 03:11:35.978651 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.978617 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r4vd\" (UniqueName: \"kubernetes.io/projected/9c520de8-a92f-4869-bfd6-4d9cd1806f8a-kube-api-access-8r4vd\") pod \"payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9\" (UID: \"9c520de8-a92f-4869-bfd6-4d9cd1806f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9" Sep 14 03:11:35.978879 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.978666 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd32c34-3a3c-46bc-a639-7a9c0916a56f-plugins-config-volume\") pod \"payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln\" (UID: \"abd32c34-3a3c-46bc-a639-7a9c0916a56f\") " pod="openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln" Sep 14 03:11:35.978879 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.978682 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv9np\" (UniqueName: \"kubernetes.io/projected/abd32c34-3a3c-46bc-a639-7a9c0916a56f-kube-api-access-lv9np\") pod \"payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln\" (UID: \"abd32c34-3a3c-46bc-a639-7a9c0916a56f\") " pod="openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln" Sep 14 03:11:35.979405 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.979380 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd32c34-3a3c-46bc-a639-7a9c0916a56f-plugins-config-volume\") pod \"payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln\" (UID: \"abd32c34-3a3c-46bc-a639-7a9c0916a56f\") " pod="openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln" Sep 14 03:11:35.979550 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.979525 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c520de8-a92f-4869-bfd6-4d9cd1806f8a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9\" (UID: \"9c520de8-a92f-4869-bfd6-4d9cd1806f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9" Sep 14 03:11:35.987036 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.987011 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv9np\" (UniqueName: \"kubernetes.io/projected/abd32c34-3a3c-46bc-a639-7a9c0916a56f-kube-api-access-lv9np\") pod \"payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln\" (UID: \"abd32c34-3a3c-46bc-a639-7a9c0916a56f\") " pod="openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln" Sep 14 03:11:35.987125 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:35.987077 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r4vd\" (UniqueName: \"kubernetes.io/projected/9c520de8-a92f-4869-bfd6-4d9cd1806f8a-kube-api-access-8r4vd\") pod \"payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9\" (UID: \"9c520de8-a92f-4869-bfd6-4d9cd1806f8a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9" Sep 14 03:11:36.127566 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:36.127536 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9" Sep 14 03:11:36.181769 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:36.181452 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln" Sep 14 03:11:36.402553 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:36.402529 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9"] Sep 14 03:11:36.404824 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:11:36.404800 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c520de8_a92f_4869_bfd6_4d9cd1806f8a.slice/crio-cc8070c074b998fb92fc0a3caa510de5a393cc5e40c6f4b37cb69b025d7e8deb WatchSource:0}: Error finding container cc8070c074b998fb92fc0a3caa510de5a393cc5e40c6f4b37cb69b025d7e8deb: Status 404 returned error can't find the container with id cc8070c074b998fb92fc0a3caa510de5a393cc5e40c6f4b37cb69b025d7e8deb Sep 14 03:11:36.464339 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:36.464305 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln"] Sep 14 03:11:36.466974 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:11:36.466945 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd32c34_3a3c_46bc_a639_7a9c0916a56f.slice/crio-e636eb2eb21c3fc614b59fa7a20c169566d32188bc467c02e8073cbec2c6066e WatchSource:0}: Error finding container e636eb2eb21c3fc614b59fa7a20c169566d32188bc467c02e8073cbec2c6066e: Status 404 returned error can't find the container with id e636eb2eb21c3fc614b59fa7a20c169566d32188bc467c02e8073cbec2c6066e Sep 14 03:11:36.933377 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:36.933341 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln" event={"ID":"abd32c34-3a3c-46bc-a639-7a9c0916a56f","Type":"ContainerStarted","Data":"2d2ff6884cb8b7970e47960c1dc7e38b781ed334166516ce3e89963b5b954a3b"} Sep 14 03:11:36.933605 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:36.933402 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln" event={"ID":"abd32c34-3a3c-46bc-a639-7a9c0916a56f","Type":"ContainerStarted","Data":"e636eb2eb21c3fc614b59fa7a20c169566d32188bc467c02e8073cbec2c6066e"} Sep 14 03:11:36.933605 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:36.933459 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln" Sep 14 03:11:36.935015 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:36.934992 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9" event={"ID":"9c520de8-a92f-4869-bfd6-4d9cd1806f8a","Type":"ContainerStarted","Data":"7cd6622d0a2fc2a95cf9d75133512c8672bf579326fc9623a8b29147adb9c861"} Sep 14 03:11:36.935120 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:36.935034 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9" event={"ID":"9c520de8-a92f-4869-bfd6-4d9cd1806f8a","Type":"ContainerStarted","Data":"cc8070c074b998fb92fc0a3caa510de5a393cc5e40c6f4b37cb69b025d7e8deb"} Sep 14 03:11:36.935120 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:36.935060 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9" Sep 14 03:11:36.951460 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:36.951418 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln" podStartSLOduration=1.9514041469999999 podStartE2EDuration="1.951404147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:11:36.950018221 +0000 UTC m=+1268.679677355" watchObservedRunningTime="2026-09-14 03:11:36.951404147 +0000 UTC m=+1268.681063324" Sep 14 03:11:36.968419 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:36.968306 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9" podStartSLOduration=1.968291658 podStartE2EDuration="1.968291658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:11:36.967509379 +0000 UTC m=+1268.697168495" watchObservedRunningTime="2026-09-14 03:11:36.968291658 +0000 UTC m=+1268.697950792" Sep 14 03:11:47.942719 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:47.942688 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln" Sep 14 03:11:47.943176 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:47.942830 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9" Sep 14 03:11:57.106346 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:57.106303 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:58.492834 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:58.492799 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:59.092964 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:59.092925 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:59.785909 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:59.785869 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:11:59.885833 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:11:59.885785 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:12:00.544527 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:12:00.544468 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:12:01.386719 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:12:01.386689 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:12:11.011526 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:12:11.011462 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:12:21.101124 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:12:21.101083 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:12:28.906954 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:12:28.906923 2564 scope.go:117] "RemoveContainer" containerID="d55cf395ae96c8ea118a6a935b0d5c12740de5d9560ac6b110285f385cc2933b" Sep 14 03:12:50.236211 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:12:50.236116 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:07.225413 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:07.225373 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:08.112211 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:08.112171 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:08.917469 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:08.917437 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:09.586676 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:09.586639 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:10.417436 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:10.417393 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:11.062454 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:11.062414 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:11.163674 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:11.163641 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:19.212950 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:19.212913 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:21.978001 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:21.977948 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:22.000561 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.000531 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln"] Sep 14 03:13:22.000789 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.000760 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln" podUID="abd32c34-3a3c-46bc-a639-7a9c0916a56f" containerName="payload-processing" containerID="cri-o://2d2ff6884cb8b7970e47960c1dc7e38b781ed334166516ce3e89963b5b954a3b" gracePeriod=30 Sep 14 03:13:22.012238 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.011839 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9"] Sep 14 03:13:22.012238 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.012076 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9" podUID="9c520de8-a92f-4869-bfd6-4d9cd1806f8a" containerName="payload-pre-processing" containerID="cri-o://7cd6622d0a2fc2a95cf9d75133512c8672bf579326fc9623a8b29147adb9c861" gracePeriod=30 Sep 14 03:13:22.049337 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.049307 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:22.123108 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.123073 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:22.224906 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.224870 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:22.300831 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.300806 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln" Sep 14 03:13:22.302858 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.302840 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9" Sep 14 03:13:22.385176 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.385148 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r4vd\" (UniqueName: \"kubernetes.io/projected/9c520de8-a92f-4869-bfd6-4d9cd1806f8a-kube-api-access-8r4vd\") pod \"9c520de8-a92f-4869-bfd6-4d9cd1806f8a\" (UID: \"9c520de8-a92f-4869-bfd6-4d9cd1806f8a\") " Sep 14 03:13:22.385336 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.385192 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c520de8-a92f-4869-bfd6-4d9cd1806f8a-plugins-config-volume\") pod \"9c520de8-a92f-4869-bfd6-4d9cd1806f8a\" (UID: \"9c520de8-a92f-4869-bfd6-4d9cd1806f8a\") " Sep 14 03:13:22.385405 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.385386 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd32c34-3a3c-46bc-a639-7a9c0916a56f-plugins-config-volume\") pod \"abd32c34-3a3c-46bc-a639-7a9c0916a56f\" (UID: \"abd32c34-3a3c-46bc-a639-7a9c0916a56f\") " Sep 14 03:13:22.385555 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.385538 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lv9np\" (UniqueName: \"kubernetes.io/projected/abd32c34-3a3c-46bc-a639-7a9c0916a56f-kube-api-access-lv9np\") pod \"abd32c34-3a3c-46bc-a639-7a9c0916a56f\" (UID: \"abd32c34-3a3c-46bc-a639-7a9c0916a56f\") " Sep 14 03:13:22.385640 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.385614 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c520de8-a92f-4869-bfd6-4d9cd1806f8a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9c520de8-a92f-4869-bfd6-4d9cd1806f8a" (UID: "9c520de8-a92f-4869-bfd6-4d9cd1806f8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:13:22.385787 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.385758 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd32c34-3a3c-46bc-a639-7a9c0916a56f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "abd32c34-3a3c-46bc-a639-7a9c0916a56f" (UID: "abd32c34-3a3c-46bc-a639-7a9c0916a56f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:13:22.385847 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.385833 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c520de8-a92f-4869-bfd6-4d9cd1806f8a-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:13:22.385891 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.385849 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd32c34-3a3c-46bc-a639-7a9c0916a56f-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:13:22.387644 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.387618 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c520de8-a92f-4869-bfd6-4d9cd1806f8a-kube-api-access-8r4vd" (OuterVolumeSpecName: "kube-api-access-8r4vd") pod "9c520de8-a92f-4869-bfd6-4d9cd1806f8a" (UID: "9c520de8-a92f-4869-bfd6-4d9cd1806f8a"). InnerVolumeSpecName "kube-api-access-8r4vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:13:22.387732 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.387659 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd32c34-3a3c-46bc-a639-7a9c0916a56f-kube-api-access-lv9np" (OuterVolumeSpecName: "kube-api-access-lv9np") pod "abd32c34-3a3c-46bc-a639-7a9c0916a56f" (UID: "abd32c34-3a3c-46bc-a639-7a9c0916a56f"). InnerVolumeSpecName "kube-api-access-lv9np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:13:22.484456 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.484422 2564 generic.go:356] "Generic (PLEG): container finished" podID="abd32c34-3a3c-46bc-a639-7a9c0916a56f" containerID="2d2ff6884cb8b7970e47960c1dc7e38b781ed334166516ce3e89963b5b954a3b" exitCode=0 Sep 14 03:13:22.484655 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.484509 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln" Sep 14 03:13:22.484655 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.484516 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln" event={"ID":"abd32c34-3a3c-46bc-a639-7a9c0916a56f","Type":"ContainerDied","Data":"2d2ff6884cb8b7970e47960c1dc7e38b781ed334166516ce3e89963b5b954a3b"} Sep 14 03:13:22.484655 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.484573 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln" event={"ID":"abd32c34-3a3c-46bc-a639-7a9c0916a56f","Type":"ContainerDied","Data":"e636eb2eb21c3fc614b59fa7a20c169566d32188bc467c02e8073cbec2c6066e"} Sep 14 03:13:22.484655 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.484595 2564 scope.go:117] "RemoveContainer" containerID="2d2ff6884cb8b7970e47960c1dc7e38b781ed334166516ce3e89963b5b954a3b" Sep 14 03:13:22.486044 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.486023 2564 generic.go:356] "Generic (PLEG): container finished" podID="9c520de8-a92f-4869-bfd6-4d9cd1806f8a" containerID="7cd6622d0a2fc2a95cf9d75133512c8672bf579326fc9623a8b29147adb9c861" exitCode=0 Sep 14 03:13:22.486148 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.486078 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9" Sep 14 03:13:22.486148 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.486129 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9" event={"ID":"9c520de8-a92f-4869-bfd6-4d9cd1806f8a","Type":"ContainerDied","Data":"7cd6622d0a2fc2a95cf9d75133512c8672bf579326fc9623a8b29147adb9c861"} Sep 14 03:13:22.486443 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.486162 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9" event={"ID":"9c520de8-a92f-4869-bfd6-4d9cd1806f8a","Type":"ContainerDied","Data":"cc8070c074b998fb92fc0a3caa510de5a393cc5e40c6f4b37cb69b025d7e8deb"} Sep 14 03:13:22.486533 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.486513 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv9np\" (UniqueName: \"kubernetes.io/projected/abd32c34-3a3c-46bc-a639-7a9c0916a56f-kube-api-access-lv9np\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:13:22.486601 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.486541 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r4vd\" (UniqueName: \"kubernetes.io/projected/9c520de8-a92f-4869-bfd6-4d9cd1806f8a-kube-api-access-8r4vd\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:13:22.494863 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.494844 2564 scope.go:117] "RemoveContainer" containerID="2d2ff6884cb8b7970e47960c1dc7e38b781ed334166516ce3e89963b5b954a3b" Sep 14 03:13:22.495241 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:13:22.495219 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d2ff6884cb8b7970e47960c1dc7e38b781ed334166516ce3e89963b5b954a3b\": container with ID starting with 2d2ff6884cb8b7970e47960c1dc7e38b781ed334166516ce3e89963b5b954a3b not found: ID does not exist" containerID="2d2ff6884cb8b7970e47960c1dc7e38b781ed334166516ce3e89963b5b954a3b" Sep 14 03:13:22.495319 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.495250 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d2ff6884cb8b7970e47960c1dc7e38b781ed334166516ce3e89963b5b954a3b"} err="failed to get container status \"2d2ff6884cb8b7970e47960c1dc7e38b781ed334166516ce3e89963b5b954a3b\": rpc error: code = NotFound desc = could not find container \"2d2ff6884cb8b7970e47960c1dc7e38b781ed334166516ce3e89963b5b954a3b\": container with ID starting with 2d2ff6884cb8b7970e47960c1dc7e38b781ed334166516ce3e89963b5b954a3b not found: ID does not exist" Sep 14 03:13:22.495319 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.495269 2564 scope.go:117] "RemoveContainer" containerID="7cd6622d0a2fc2a95cf9d75133512c8672bf579326fc9623a8b29147adb9c861" Sep 14 03:13:22.504006 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.503987 2564 scope.go:117] "RemoveContainer" containerID="7cd6622d0a2fc2a95cf9d75133512c8672bf579326fc9623a8b29147adb9c861" Sep 14 03:13:22.504312 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:13:22.504292 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd6622d0a2fc2a95cf9d75133512c8672bf579326fc9623a8b29147adb9c861\": container with ID starting with 7cd6622d0a2fc2a95cf9d75133512c8672bf579326fc9623a8b29147adb9c861 not found: ID does not exist" containerID="7cd6622d0a2fc2a95cf9d75133512c8672bf579326fc9623a8b29147adb9c861" Sep 14 03:13:22.504362 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.504318 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd6622d0a2fc2a95cf9d75133512c8672bf579326fc9623a8b29147adb9c861"} err="failed to get container status \"7cd6622d0a2fc2a95cf9d75133512c8672bf579326fc9623a8b29147adb9c861\": rpc error: code = NotFound desc = could not find container \"7cd6622d0a2fc2a95cf9d75133512c8672bf579326fc9623a8b29147adb9c861\": container with ID starting with 7cd6622d0a2fc2a95cf9d75133512c8672bf579326fc9623a8b29147adb9c861 not found: ID does not exist" Sep 14 03:13:22.519437 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.519376 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln"] Sep 14 03:13:22.522724 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.522702 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d6ac6dc8-767c678569-dt4ln"] Sep 14 03:13:22.535527 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.535503 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9"] Sep 14 03:13:22.538763 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.538741 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d6ac6dc8-7c7f565974-gpnr9"] Sep 14 03:13:22.662275 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.662244 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:22.750760 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.750729 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c520de8-a92f-4869-bfd6-4d9cd1806f8a" path="/var/lib/kubelet/pods/9c520de8-a92f-4869-bfd6-4d9cd1806f8a/volumes" Sep 14 03:13:22.751125 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:22.751111 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abd32c34-3a3c-46bc-a639-7a9c0916a56f" path="/var/lib/kubelet/pods/abd32c34-3a3c-46bc-a639-7a9c0916a56f/volumes" Sep 14 03:13:23.228217 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:23.228182 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:23.353369 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:23.353276 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:24.072246 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:24.072212 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:24.915251 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:24.915217 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:25.370470 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:25.370370 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:25.618463 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:25.618429 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:28.622269 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:28.622237 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:29.518018 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:29.517984 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:30.444060 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:30.444022 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:48.229667 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:48.229632 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:50.428722 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:50.428667 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:13:53.751642 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:13:53.751602 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:01.161497 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:01.161434 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:07.565735 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:07.565695 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:08.434017 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:08.433956 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:14.235248 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:14.235209 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:14.336001 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:14.335963 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:14.511426 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:14.511331 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:15.314938 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:15.314900 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:15.735354 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:15.735314 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:15.835577 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:15.835544 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:15.935178 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:15.935142 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:16.617887 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:16.617850 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:16.757851 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:16.757818 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:18.033153 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:18.033116 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:24.194548 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:24.194442 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:24.272467 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:24.272431 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:24.871312 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:24.871271 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:30.855994 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:30.855953 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:30.922857 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:30.922828 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:31.029963 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:31.029926 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:38.069348 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:38.069314 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:38.135102 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:38.135064 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:41.534225 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:41.534186 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:14:41.752519 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:14:41.752453 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:00.182319 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:00.182282 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0c919e-29822595-fpqkj"] Sep 14 03:15:00.182957 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:00.182924 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c520de8-a92f-4869-bfd6-4d9cd1806f8a" containerName="payload-pre-processing" Sep 14 03:15:00.182957 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:00.182945 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c520de8-a92f-4869-bfd6-4d9cd1806f8a" containerName="payload-pre-processing" Sep 14 03:15:00.183090 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:00.182969 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abd32c34-3a3c-46bc-a639-7a9c0916a56f" containerName="payload-processing" Sep 14 03:15:00.183090 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:00.182978 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="abd32c34-3a3c-46bc-a639-7a9c0916a56f" containerName="payload-processing" Sep 14 03:15:00.183190 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:00.183091 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="abd32c34-3a3c-46bc-a639-7a9c0916a56f" containerName="payload-processing" Sep 14 03:15:00.183190 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:00.183107 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c520de8-a92f-4869-bfd6-4d9cd1806f8a" containerName="payload-pre-processing" Sep 14 03:15:00.186042 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:00.186021 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f77a9-29822595-fqczj"] Sep 14 03:15:00.186186 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:00.186166 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0c919e-29822595-fpqkj" Sep 14 03:15:00.189538 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:00.189517 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4hf5p\"" Sep 14 03:15:00.190199 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:00.190182 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f77a9-29822595-fqczj" Sep 14 03:15:00.198857 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:00.198834 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0c919e-29822595-fpqkj" Sep 14 03:15:00.199177 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:00.199153 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f77a9-29822595-fqczj"] Sep 14 03:15:00.202136 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:00.202007 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0c919e-29822595-fpqkj"] Sep 14 03:15:00.207767 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:00.207745 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f77a9-29822595-fqczj" Sep 14 03:15:00.482813 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:00.482781 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f77a9-29822595-fqczj"] Sep 14 03:15:00.484909 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:15:00.484878 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a8a1eb9_90ad_41f6_9236_f51d2fc75e54.slice/crio-65c624fd21dd5885c43b9718ccd116a8444af1643312dafdc3f91a731790f6c9 WatchSource:0}: Error finding container 65c624fd21dd5885c43b9718ccd116a8444af1643312dafdc3f91a731790f6c9: Status 404 returned error can't find the container with id 65c624fd21dd5885c43b9718ccd116a8444af1643312dafdc3f91a731790f6c9 Sep 14 03:15:00.485033 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:00.484981 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0c919e-29822595-fpqkj"] Sep 14 03:15:00.486731 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:15:00.486705 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6935df75_6d19_4636_b2a4_354b1c1bdd58.slice/crio-edc52557ea1ef272e15c6c88ecddf1a2cd4f84b521340ad993e1eed37f739973 WatchSource:0}: Error finding container edc52557ea1ef272e15c6c88ecddf1a2cd4f84b521340ad993e1eed37f739973: Status 404 returned error can't find the container with id edc52557ea1ef272e15c6c88ecddf1a2cd4f84b521340ad993e1eed37f739973 Sep 14 03:15:01.019572 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:01.019533 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0c919e-29822595-fpqkj" event={"ID":"6935df75-6d19-4636-b2a4-354b1c1bdd58","Type":"ContainerStarted","Data":"67b0c0c1c99f4a5e623dd34f3eb09a5c1e278b8c84b2b4df4827dc62c93b2ce6"} Sep 14 03:15:01.019769 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:01.019596 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0c919e-29822595-fpqkj" event={"ID":"6935df75-6d19-4636-b2a4-354b1c1bdd58","Type":"ContainerStarted","Data":"edc52557ea1ef272e15c6c88ecddf1a2cd4f84b521340ad993e1eed37f739973"} Sep 14 03:15:01.021071 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:01.021052 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f77a9-29822595-fqczj" event={"ID":"4a8a1eb9-90ad-41f6-9236-f51d2fc75e54","Type":"ContainerStarted","Data":"baf851d1b62b3192989d73d2a92095ecf9b9921c0c8eef3822a669a9cb309eca"} Sep 14 03:15:01.021165 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:01.021088 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f77a9-29822595-fqczj" event={"ID":"4a8a1eb9-90ad-41f6-9236-f51d2fc75e54","Type":"ContainerStarted","Data":"65c624fd21dd5885c43b9718ccd116a8444af1643312dafdc3f91a731790f6c9"} Sep 14 03:15:01.050966 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:01.050875 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f77a9-29822595-fqczj" podStartSLOduration=1.050860816 podStartE2EDuration="1.050860816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:15:01.049621708 +0000 UTC m=+1472.779280824" watchObservedRunningTime="2026-09-14 03:15:01.050860816 +0000 UTC m=+1472.780519933" Sep 14 03:15:01.051709 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:01.051674 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0c919e-29822595-fpqkj" podStartSLOduration=1.05166687 podStartE2EDuration="1.05166687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:15:01.035409626 +0000 UTC m=+1472.765068745" watchObservedRunningTime="2026-09-14 03:15:01.05166687 +0000 UTC m=+1472.781325987" Sep 14 03:15:01.110787 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:01.110742 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:02.028863 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:02.028837 2564 generic.go:356] "Generic (PLEG): container finished" podID="6935df75-6d19-4636-b2a4-354b1c1bdd58" containerID="67b0c0c1c99f4a5e623dd34f3eb09a5c1e278b8c84b2b4df4827dc62c93b2ce6" exitCode=0 Sep 14 03:15:02.029317 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:02.028926 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0c919e-29822595-fpqkj" event={"ID":"6935df75-6d19-4636-b2a4-354b1c1bdd58","Type":"ContainerDied","Data":"67b0c0c1c99f4a5e623dd34f3eb09a5c1e278b8c84b2b4df4827dc62c93b2ce6"} Sep 14 03:15:02.030610 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:02.030585 2564 generic.go:356] "Generic (PLEG): container finished" podID="4a8a1eb9-90ad-41f6-9236-f51d2fc75e54" containerID="baf851d1b62b3192989d73d2a92095ecf9b9921c0c8eef3822a669a9cb309eca" exitCode=0 Sep 14 03:15:02.030711 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:02.030652 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f77a9-29822595-fqczj" event={"ID":"4a8a1eb9-90ad-41f6-9236-f51d2fc75e54","Type":"ContainerDied","Data":"baf851d1b62b3192989d73d2a92095ecf9b9921c0c8eef3822a669a9cb309eca"} Sep 14 03:15:02.108648 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:02.108617 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:02.555411 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:02.555374 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:02.712177 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:02.712145 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:02.903726 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:02.903680 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:03.042945 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:03.042914 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:03.176214 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:03.176186 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f77a9-29822595-fqczj" Sep 14 03:15:03.216748 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:03.216719 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0c919e-29822595-fpqkj" Sep 14 03:15:04.042166 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:04.042140 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f77a9-29822595-fqczj" Sep 14 03:15:04.042336 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:04.042186 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f77a9-29822595-fqczj" event={"ID":"4a8a1eb9-90ad-41f6-9236-f51d2fc75e54","Type":"ContainerDied","Data":"65c624fd21dd5885c43b9718ccd116a8444af1643312dafdc3f91a731790f6c9"} Sep 14 03:15:04.042336 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:04.042240 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65c624fd21dd5885c43b9718ccd116a8444af1643312dafdc3f91a731790f6c9" Sep 14 03:15:04.043585 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:04.043568 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0c919e-29822595-fpqkj" Sep 14 03:15:04.043915 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:04.043581 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0c919e-29822595-fpqkj" event={"ID":"6935df75-6d19-4636-b2a4-354b1c1bdd58","Type":"ContainerDied","Data":"edc52557ea1ef272e15c6c88ecddf1a2cd4f84b521340ad993e1eed37f739973"} Sep 14 03:15:04.043915 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:04.043624 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edc52557ea1ef272e15c6c88ecddf1a2cd4f84b521340ad993e1eed37f739973" Sep 14 03:15:06.338499 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:06.338438 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:06.486591 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:06.486558 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:06.585663 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:06.585629 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:07.244588 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:07.244554 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:08.139084 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:08.139049 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:08.824904 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:08.824869 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:09.520894 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:09.520857 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:09.619622 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:09.619589 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:09.717685 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:09.717647 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:09.820865 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:09.820784 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:09.919720 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:09.919690 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:10.149875 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.149837 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:10.241158 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.241120 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:10.339503 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.339433 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9"] Sep 14 03:15:10.339882 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.339864 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a8a1eb9-90ad-41f6-9236-f51d2fc75e54" containerName="cleanup" Sep 14 03:15:10.339882 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.339879 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a8a1eb9-90ad-41f6-9236-f51d2fc75e54" containerName="cleanup" Sep 14 03:15:10.340049 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.339898 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6935df75-6d19-4636-b2a4-354b1c1bdd58" containerName="cleanup" Sep 14 03:15:10.340049 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.339903 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6935df75-6d19-4636-b2a4-354b1c1bdd58" containerName="cleanup" Sep 14 03:15:10.340049 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.339961 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6935df75-6d19-4636-b2a4-354b1c1bdd58" containerName="cleanup" Sep 14 03:15:10.340049 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.339969 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a8a1eb9-90ad-41f6-9236-f51d2fc75e54" containerName="cleanup" Sep 14 03:15:10.343288 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.343268 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9" Sep 14 03:15:10.345834 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.345818 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2611af\"" Sep 14 03:15:10.345916 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.345818 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2611af-dockercfg-qcx92\"" Sep 14 03:15:10.357786 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.357756 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9"] Sep 14 03:15:10.372445 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.372233 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02aaca25-da12-4173-adc6-a71fdfd95346-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9\" (UID: \"02aaca25-da12-4173-adc6-a71fdfd95346\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9" Sep 14 03:15:10.372445 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.372340 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chsbq\" (UniqueName: \"kubernetes.io/projected/02aaca25-da12-4173-adc6-a71fdfd95346-kube-api-access-chsbq\") pod \"payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9\" (UID: \"02aaca25-da12-4173-adc6-a71fdfd95346\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9" Sep 14 03:15:10.398226 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.398195 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9"] Sep 14 03:15:10.401715 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.401671 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9" Sep 14 03:15:10.414217 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.414194 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9"] Sep 14 03:15:10.473926 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.473890 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chsbq\" (UniqueName: \"kubernetes.io/projected/02aaca25-da12-4173-adc6-a71fdfd95346-kube-api-access-chsbq\") pod \"payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9\" (UID: \"02aaca25-da12-4173-adc6-a71fdfd95346\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9" Sep 14 03:15:10.474082 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.473977 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsv7g\" (UniqueName: \"kubernetes.io/projected/3fada7bb-85e2-4680-ab7b-e7dde698eca3-kube-api-access-fsv7g\") pod \"payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9\" (UID: \"3fada7bb-85e2-4680-ab7b-e7dde698eca3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9" Sep 14 03:15:10.474082 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.474068 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02aaca25-da12-4173-adc6-a71fdfd95346-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9\" (UID: \"02aaca25-da12-4173-adc6-a71fdfd95346\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9" Sep 14 03:15:10.474195 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.474166 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fada7bb-85e2-4680-ab7b-e7dde698eca3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9\" (UID: \"3fada7bb-85e2-4680-ab7b-e7dde698eca3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9" Sep 14 03:15:10.474718 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.474701 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02aaca25-da12-4173-adc6-a71fdfd95346-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9\" (UID: \"02aaca25-da12-4173-adc6-a71fdfd95346\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9" Sep 14 03:15:10.482735 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.482713 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chsbq\" (UniqueName: \"kubernetes.io/projected/02aaca25-da12-4173-adc6-a71fdfd95346-kube-api-access-chsbq\") pod \"payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9\" (UID: \"02aaca25-da12-4173-adc6-a71fdfd95346\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9" Sep 14 03:15:10.575566 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.575533 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsv7g\" (UniqueName: \"kubernetes.io/projected/3fada7bb-85e2-4680-ab7b-e7dde698eca3-kube-api-access-fsv7g\") pod \"payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9\" (UID: \"3fada7bb-85e2-4680-ab7b-e7dde698eca3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9" Sep 14 03:15:10.575932 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.575589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fada7bb-85e2-4680-ab7b-e7dde698eca3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9\" (UID: \"3fada7bb-85e2-4680-ab7b-e7dde698eca3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9" Sep 14 03:15:10.576185 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.576167 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fada7bb-85e2-4680-ab7b-e7dde698eca3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9\" (UID: \"3fada7bb-85e2-4680-ab7b-e7dde698eca3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9" Sep 14 03:15:10.585571 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.585539 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsv7g\" (UniqueName: \"kubernetes.io/projected/3fada7bb-85e2-4680-ab7b-e7dde698eca3-kube-api-access-fsv7g\") pod \"payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9\" (UID: \"3fada7bb-85e2-4680-ab7b-e7dde698eca3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9" Sep 14 03:15:10.654135 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.654084 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9" Sep 14 03:15:10.715721 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.715270 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9" Sep 14 03:15:10.869113 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.869082 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:10.928846 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:10.928822 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9"] Sep 14 03:15:10.931601 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:15:10.931577 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02aaca25_da12_4173_adc6_a71fdfd95346.slice/crio-2429c0853e30edc4b7b1ce6ebe17c86e697502fb58a72cab9962ce9e45caa766 WatchSource:0}: Error finding container 2429c0853e30edc4b7b1ce6ebe17c86e697502fb58a72cab9962ce9e45caa766: Status 404 returned error can't find the container with id 2429c0853e30edc4b7b1ce6ebe17c86e697502fb58a72cab9962ce9e45caa766 Sep 14 03:15:11.004915 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:11.004889 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9"] Sep 14 03:15:11.007536 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:15:11.007512 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fada7bb_85e2_4680_ab7b_e7dde698eca3.slice/crio-7a258e272caa4eb8a2af50de03aa05a8c453fc96e9ef8f0b3ab75e323f9a2be9 WatchSource:0}: Error finding container 7a258e272caa4eb8a2af50de03aa05a8c453fc96e9ef8f0b3ab75e323f9a2be9: Status 404 returned error can't find the container with id 7a258e272caa4eb8a2af50de03aa05a8c453fc96e9ef8f0b3ab75e323f9a2be9 Sep 14 03:15:11.086096 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:11.086057 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9" event={"ID":"3fada7bb-85e2-4680-ab7b-e7dde698eca3","Type":"ContainerStarted","Data":"3a93beaadf6409b642bbd8dc54d5cf9605393cf94a8da27bcc96ae2982600c52"} Sep 14 03:15:11.086234 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:11.086136 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9" event={"ID":"3fada7bb-85e2-4680-ab7b-e7dde698eca3","Type":"ContainerStarted","Data":"7a258e272caa4eb8a2af50de03aa05a8c453fc96e9ef8f0b3ab75e323f9a2be9"} Sep 14 03:15:11.086234 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:11.086168 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9" Sep 14 03:15:11.088052 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:11.088022 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9" event={"ID":"02aaca25-da12-4173-adc6-a71fdfd95346","Type":"ContainerStarted","Data":"30037c5af164f8f27fd9a5f6c860e5b02f0017edc3b115b7f0709d723e676c60"} Sep 14 03:15:11.088184 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:11.088080 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9" event={"ID":"02aaca25-da12-4173-adc6-a71fdfd95346","Type":"ContainerStarted","Data":"2429c0853e30edc4b7b1ce6ebe17c86e697502fb58a72cab9962ce9e45caa766"} Sep 14 03:15:11.088184 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:11.088115 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9" Sep 14 03:15:11.106821 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:11.106775 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9" podStartSLOduration=1.106761533 podStartE2EDuration="1.106761533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:15:11.104432584 +0000 UTC m=+1482.834091701" watchObservedRunningTime="2026-09-14 03:15:11.106761533 +0000 UTC m=+1482.836420678" Sep 14 03:15:11.123027 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:11.122976 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9" podStartSLOduration=1.122956825 podStartE2EDuration="1.122956825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:15:11.121667971 +0000 UTC m=+1482.851327087" watchObservedRunningTime="2026-09-14 03:15:11.122956825 +0000 UTC m=+1482.852615953" Sep 14 03:15:13.425316 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:13.425278 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:13.535681 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:13.535643 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:13.625575 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:13.625531 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:14.280399 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:14.280354 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:15.055162 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:15.055124 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:15.815718 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:15.815685 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:16.522163 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:16.522129 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:17.346000 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:17.345959 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:17.431332 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:17.431299 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:17.836830 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:17.836794 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:17.936419 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:17.936384 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:17.961392 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:17.961364 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55"] Sep 14 03:15:17.965594 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:17.965572 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55" Sep 14 03:15:17.967976 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:17.967955 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-40c586\"" Sep 14 03:15:17.967976 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:17.967969 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-40c586-dockercfg-zzk5z\"" Sep 14 03:15:17.973636 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:17.973613 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55"] Sep 14 03:15:18.042959 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:18.042929 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnv62\" (UniqueName: \"kubernetes.io/projected/51e4c384-415f-43ca-a05d-6e3d1f5ee5f3-kube-api-access-fnv62\") pod \"payload-processing-e2e-api-a-40c586-c868cd87-nfp55\" (UID: \"51e4c384-415f-43ca-a05d-6e3d1f5ee5f3\") " pod="openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55" Sep 14 03:15:18.043128 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:18.043004 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e4c384-415f-43ca-a05d-6e3d1f5ee5f3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-40c586-c868cd87-nfp55\" (UID: \"51e4c384-415f-43ca-a05d-6e3d1f5ee5f3\") " pod="openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55" Sep 14 03:15:18.049240 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:18.049204 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:18.144280 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:18.144255 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e4c384-415f-43ca-a05d-6e3d1f5ee5f3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-40c586-c868cd87-nfp55\" (UID: \"51e4c384-415f-43ca-a05d-6e3d1f5ee5f3\") " pod="openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55" Sep 14 03:15:18.144398 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:18.144343 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnv62\" (UniqueName: \"kubernetes.io/projected/51e4c384-415f-43ca-a05d-6e3d1f5ee5f3-kube-api-access-fnv62\") pod \"payload-processing-e2e-api-a-40c586-c868cd87-nfp55\" (UID: \"51e4c384-415f-43ca-a05d-6e3d1f5ee5f3\") " pod="openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55" Sep 14 03:15:18.144959 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:18.144939 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e4c384-415f-43ca-a05d-6e3d1f5ee5f3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-40c586-c868cd87-nfp55\" (UID: \"51e4c384-415f-43ca-a05d-6e3d1f5ee5f3\") " pod="openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55" Sep 14 03:15:18.152778 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:18.152750 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnv62\" (UniqueName: \"kubernetes.io/projected/51e4c384-415f-43ca-a05d-6e3d1f5ee5f3-kube-api-access-fnv62\") pod \"payload-processing-e2e-api-a-40c586-c868cd87-nfp55\" (UID: \"51e4c384-415f-43ca-a05d-6e3d1f5ee5f3\") " pod="openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55" Sep 14 03:15:18.279453 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:18.279422 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55" Sep 14 03:15:18.552137 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:18.552111 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55"] Sep 14 03:15:18.554280 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:15:18.554249 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e4c384_415f_43ca_a05d_6e3d1f5ee5f3.slice/crio-97b6b9750e54315c48ff0cfd37f0f89a3f860ab93ddfb763e8c445e228f13859 WatchSource:0}: Error finding container 97b6b9750e54315c48ff0cfd37f0f89a3f860ab93ddfb763e8c445e228f13859: Status 404 returned error can't find the container with id 97b6b9750e54315c48ff0cfd37f0f89a3f860ab93ddfb763e8c445e228f13859 Sep 14 03:15:19.144905 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:19.144858 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55" event={"ID":"51e4c384-415f-43ca-a05d-6e3d1f5ee5f3","Type":"ContainerStarted","Data":"b562449bc044f65e3c6370ed867c0946ed9610677bcdde4aed647ecb6ee250e3"} Sep 14 03:15:19.145362 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:19.144935 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55" event={"ID":"51e4c384-415f-43ca-a05d-6e3d1f5ee5f3","Type":"ContainerStarted","Data":"97b6b9750e54315c48ff0cfd37f0f89a3f860ab93ddfb763e8c445e228f13859"} Sep 14 03:15:19.145362 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:19.144968 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55" Sep 14 03:15:22.095436 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:22.095406 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9" Sep 14 03:15:22.095820 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:22.095605 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9" Sep 14 03:15:22.113430 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:22.113384 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55" podStartSLOduration=5.113369383 podStartE2EDuration="5.113369383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:15:19.167778042 +0000 UTC m=+1490.897437159" watchObservedRunningTime="2026-09-14 03:15:22.113369383 +0000 UTC m=+1493.843028499" Sep 14 03:15:28.850637 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:28.850612 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bwv6_4369115a-4032-4f5a-a3e4-c02efe665617/ovn-acl-logging/0.log" Sep 14 03:15:28.854106 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:28.854088 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bwv6_4369115a-4032-4f5a-a3e4-c02efe665617/ovn-acl-logging/0.log" Sep 14 03:15:30.151503 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:30.151437 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55" Sep 14 03:15:30.709163 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:30.709122 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:32.021654 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:32.021616 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:32.634300 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:32.634263 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:33.374161 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:33.374122 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:34.114557 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:34.114517 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:34.830109 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:34.830071 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:35.616928 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:35.616892 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:36.407979 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:36.407947 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:36.535557 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:36.535519 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:37.073785 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:37.073746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:37.204011 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:37.203981 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:37.548107 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:37.548065 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:37.926020 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:37.925987 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:38.023455 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:38.023425 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:38.476226 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:38.476189 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:38.719557 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:38.719525 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:38.912869 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:38.912841 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:39.274800 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:39.274704 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:39.632037 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:39.631939 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:39.733345 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:39.733318 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:40.567901 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:40.567865 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:40.694146 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:40.694099 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:40.795004 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:40.794973 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:41.523207 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:41.523177 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:42.527501 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:42.527442 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:42.747500 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:42.747435 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:42.847217 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:42.847131 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:43.372994 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:43.372959 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:44.240862 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:44.240830 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:44.341214 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:44.341178 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:44.694855 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:44.694819 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:44.793670 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:44.793628 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:44.874896 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:44.874862 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:45.109590 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:45.109468 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:45.652553 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:45.652519 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:46.626414 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:46.626383 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:47.360960 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:47.360923 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:47.459226 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:47.459189 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:47.556754 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:47.556718 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:47.839952 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:47.839916 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:47.926736 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:47.926697 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:47.971271 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:47.971245 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629"] Sep 14 03:15:47.975882 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:47.975865 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629" Sep 14 03:15:47.978012 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:47.977991 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-13d31c\"" Sep 14 03:15:47.978134 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:47.978099 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-13d31c-dockercfg-gvdfw\"" Sep 14 03:15:47.985521 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:47.985499 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629"] Sep 14 03:15:48.016507 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:48.016452 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c137e1f-0e2e-476d-8435-a0f4218569ed-plugins-config-volume\") pod \"payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629\" (UID: \"3c137e1f-0e2e-476d-8435-a0f4218569ed\") " pod="openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629" Sep 14 03:15:48.016612 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:48.016528 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcd8l\" (UniqueName: \"kubernetes.io/projected/3c137e1f-0e2e-476d-8435-a0f4218569ed-kube-api-access-hcd8l\") pod \"payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629\" (UID: \"3c137e1f-0e2e-476d-8435-a0f4218569ed\") " pod="openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629" Sep 14 03:15:48.019912 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:48.019890 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb"] Sep 14 03:15:48.023647 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:48.023631 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb" Sep 14 03:15:48.034151 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:48.034129 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb"] Sep 14 03:15:48.117870 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:48.117793 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fx8l\" (UniqueName: \"kubernetes.io/projected/014f38ab-a047-408b-9b4f-78638159e65b-kube-api-access-8fx8l\") pod \"payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb\" (UID: \"014f38ab-a047-408b-9b4f-78638159e65b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb" Sep 14 03:15:48.117870 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:48.117850 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/014f38ab-a047-408b-9b4f-78638159e65b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb\" (UID: \"014f38ab-a047-408b-9b4f-78638159e65b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb" Sep 14 03:15:48.118090 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:48.117902 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c137e1f-0e2e-476d-8435-a0f4218569ed-plugins-config-volume\") pod \"payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629\" (UID: \"3c137e1f-0e2e-476d-8435-a0f4218569ed\") " pod="openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629" Sep 14 03:15:48.118090 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:48.117946 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcd8l\" (UniqueName: \"kubernetes.io/projected/3c137e1f-0e2e-476d-8435-a0f4218569ed-kube-api-access-hcd8l\") pod \"payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629\" (UID: \"3c137e1f-0e2e-476d-8435-a0f4218569ed\") " pod="openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629" Sep 14 03:15:48.118644 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:48.118621 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c137e1f-0e2e-476d-8435-a0f4218569ed-plugins-config-volume\") pod \"payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629\" (UID: \"3c137e1f-0e2e-476d-8435-a0f4218569ed\") " pod="openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629" Sep 14 03:15:48.125760 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:48.125743 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcd8l\" (UniqueName: \"kubernetes.io/projected/3c137e1f-0e2e-476d-8435-a0f4218569ed-kube-api-access-hcd8l\") pod \"payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629\" (UID: \"3c137e1f-0e2e-476d-8435-a0f4218569ed\") " pod="openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629" Sep 14 03:15:48.218649 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:48.218622 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fx8l\" (UniqueName: \"kubernetes.io/projected/014f38ab-a047-408b-9b4f-78638159e65b-kube-api-access-8fx8l\") pod \"payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb\" (UID: \"014f38ab-a047-408b-9b4f-78638159e65b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb" Sep 14 03:15:48.218758 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:48.218652 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/014f38ab-a047-408b-9b4f-78638159e65b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb\" (UID: \"014f38ab-a047-408b-9b4f-78638159e65b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb" Sep 14 03:15:48.219276 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:48.219259 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/014f38ab-a047-408b-9b4f-78638159e65b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb\" (UID: \"014f38ab-a047-408b-9b4f-78638159e65b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb" Sep 14 03:15:48.226408 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:48.226383 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fx8l\" (UniqueName: \"kubernetes.io/projected/014f38ab-a047-408b-9b4f-78638159e65b-kube-api-access-8fx8l\") pod \"payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb\" (UID: \"014f38ab-a047-408b-9b4f-78638159e65b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb" Sep 14 03:15:48.233647 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:48.233592 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:48.286559 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:48.286535 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629" Sep 14 03:15:48.335370 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:48.334923 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb" Sep 14 03:15:48.566727 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:48.566705 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629"] Sep 14 03:15:48.568090 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:15:48.568064 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c137e1f_0e2e_476d_8435_a0f4218569ed.slice/crio-91edf6eadac502f7744eaf753cec1296725fbdd4ef459def9cad3e5d1aada3c4 WatchSource:0}: Error finding container 91edf6eadac502f7744eaf753cec1296725fbdd4ef459def9cad3e5d1aada3c4: Status 404 returned error can't find the container with id 91edf6eadac502f7744eaf753cec1296725fbdd4ef459def9cad3e5d1aada3c4 Sep 14 03:15:48.622715 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:48.622693 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb"] Sep 14 03:15:48.626503 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:15:48.626455 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014f38ab_a047_408b_9b4f_78638159e65b.slice/crio-98609046d5db15d335b9ecaf37d00e2b53aae6fc55a501ac949f08e1bc6e050b WatchSource:0}: Error finding container 98609046d5db15d335b9ecaf37d00e2b53aae6fc55a501ac949f08e1bc6e050b: Status 404 returned error can't find the container with id 98609046d5db15d335b9ecaf37d00e2b53aae6fc55a501ac949f08e1bc6e050b Sep 14 03:15:49.011831 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:49.011750 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:49.309178 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:49.309088 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629" event={"ID":"3c137e1f-0e2e-476d-8435-a0f4218569ed","Type":"ContainerStarted","Data":"0a6675abf90fbfc85bdbc35fa30c8534a5fd28b4e2100bdf74325715d6929f45"} Sep 14 03:15:49.309338 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:49.309252 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629" Sep 14 03:15:49.309338 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:49.309300 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629" event={"ID":"3c137e1f-0e2e-476d-8435-a0f4218569ed","Type":"ContainerStarted","Data":"91edf6eadac502f7744eaf753cec1296725fbdd4ef459def9cad3e5d1aada3c4"} Sep 14 03:15:49.310750 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:49.310718 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb" event={"ID":"014f38ab-a047-408b-9b4f-78638159e65b","Type":"ContainerStarted","Data":"0bf2049b553767efb65ed7bc3f063bd97559dbb5b3a0d019905671942ed1fd93"} Sep 14 03:15:49.310750 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:49.310752 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb" event={"ID":"014f38ab-a047-408b-9b4f-78638159e65b","Type":"ContainerStarted","Data":"98609046d5db15d335b9ecaf37d00e2b53aae6fc55a501ac949f08e1bc6e050b"} Sep 14 03:15:49.310938 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:49.310855 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb" Sep 14 03:15:49.331591 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:49.331542 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629" podStartSLOduration=2.331528669 podStartE2EDuration="2.331528669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:15:49.329828043 +0000 UTC m=+1521.059487160" watchObservedRunningTime="2026-09-14 03:15:49.331528669 +0000 UTC m=+1521.061187799" Sep 14 03:15:49.803731 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:49.803696 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:50.515290 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:50.515258 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:15:51.215587 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:15:51.215552 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:00.319343 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:00.319310 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629" Sep 14 03:16:00.319851 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:00.319538 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb" Sep 14 03:16:00.338346 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:00.338298 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb" podStartSLOduration=12.338282949 podStartE2EDuration="12.338282949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:15:49.352389826 +0000 UTC m=+1521.082048941" watchObservedRunningTime="2026-09-14 03:16:00.338282949 +0000 UTC m=+1532.067942064" Sep 14 03:16:05.148554 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:05.148517 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:06.508542 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:06.508506 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:07.320407 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:07.320369 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:08.005604 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:08.005566 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:08.720442 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:08.720408 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:09.307222 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:09.307188 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:09.944109 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:09.944075 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:10.774906 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:10.774866 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:10.877896 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:10.877865 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:11.629448 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:11.629408 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:12.269378 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:12.269348 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:12.770415 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:12.770386 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:12.989297 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:12.989260 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:13.826180 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:13.826146 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:15.283589 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:15.283548 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:15.384251 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:15.384198 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:16.210253 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:16.210218 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:16.852261 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:16.852225 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:17.644548 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:17.644510 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:18.543847 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:18.543810 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:19.233666 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:19.233626 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:19.917742 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:19.917699 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:20.593628 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:20.593594 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:21.275996 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:21.275939 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:22.021284 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:22.021250 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:25.677589 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:25.677554 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:27.666561 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:27.666517 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:31.372906 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:31.372869 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:31.418463 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:31.418429 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55"] Sep 14 03:16:31.419138 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:31.418989 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55" podUID="51e4c384-415f-43ca-a05d-6e3d1f5ee5f3" containerName="payload-processing" containerID="cri-o://b562449bc044f65e3c6370ed867c0946ed9610677bcdde4aed647ecb6ee250e3" gracePeriod=30 Sep 14 03:16:31.466240 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:31.466206 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:31.534331 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:31.534303 2564 generic.go:356] "Generic (PLEG): container finished" podID="51e4c384-415f-43ca-a05d-6e3d1f5ee5f3" containerID="b562449bc044f65e3c6370ed867c0946ed9610677bcdde4aed647ecb6ee250e3" exitCode=0 Sep 14 03:16:31.534459 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:31.534343 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55" event={"ID":"51e4c384-415f-43ca-a05d-6e3d1f5ee5f3","Type":"ContainerDied","Data":"b562449bc044f65e3c6370ed867c0946ed9610677bcdde4aed647ecb6ee250e3"} Sep 14 03:16:31.565428 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:31.565389 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:31.665187 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:31.665149 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:31.667207 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:31.667186 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55" Sep 14 03:16:31.726357 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:31.726328 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e4c384-415f-43ca-a05d-6e3d1f5ee5f3-plugins-config-volume\") pod \"51e4c384-415f-43ca-a05d-6e3d1f5ee5f3\" (UID: \"51e4c384-415f-43ca-a05d-6e3d1f5ee5f3\") " Sep 14 03:16:31.726567 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:31.726389 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnv62\" (UniqueName: \"kubernetes.io/projected/51e4c384-415f-43ca-a05d-6e3d1f5ee5f3-kube-api-access-fnv62\") pod \"51e4c384-415f-43ca-a05d-6e3d1f5ee5f3\" (UID: \"51e4c384-415f-43ca-a05d-6e3d1f5ee5f3\") " Sep 14 03:16:31.726779 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:31.726753 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e4c384-415f-43ca-a05d-6e3d1f5ee5f3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "51e4c384-415f-43ca-a05d-6e3d1f5ee5f3" (UID: "51e4c384-415f-43ca-a05d-6e3d1f5ee5f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:16:31.728622 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:31.728604 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e4c384-415f-43ca-a05d-6e3d1f5ee5f3-kube-api-access-fnv62" (OuterVolumeSpecName: "kube-api-access-fnv62") pod "51e4c384-415f-43ca-a05d-6e3d1f5ee5f3" (UID: "51e4c384-415f-43ca-a05d-6e3d1f5ee5f3"). InnerVolumeSpecName "kube-api-access-fnv62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:16:31.828621 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:31.828571 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnv62\" (UniqueName: \"kubernetes.io/projected/51e4c384-415f-43ca-a05d-6e3d1f5ee5f3-kube-api-access-fnv62\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:16:31.828621 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:31.828614 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e4c384-415f-43ca-a05d-6e3d1f5ee5f3-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:16:31.972038 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:31.971956 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:32.069625 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:32.069594 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:32.540969 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:32.540938 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55" Sep 14 03:16:32.541375 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:32.540989 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55" event={"ID":"51e4c384-415f-43ca-a05d-6e3d1f5ee5f3","Type":"ContainerDied","Data":"97b6b9750e54315c48ff0cfd37f0f89a3f860ab93ddfb763e8c445e228f13859"} Sep 14 03:16:32.541375 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:32.541048 2564 scope.go:117] "RemoveContainer" containerID="b562449bc044f65e3c6370ed867c0946ed9610677bcdde4aed647ecb6ee250e3" Sep 14 03:16:32.567230 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:32.567201 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55"] Sep 14 03:16:32.575952 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:32.575920 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-40c586-c868cd87-nfp55"] Sep 14 03:16:32.749765 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:32.749735 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51e4c384-415f-43ca-a05d-6e3d1f5ee5f3" path="/var/lib/kubelet/pods/51e4c384-415f-43ca-a05d-6e3d1f5ee5f3/volumes" Sep 14 03:16:32.890245 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:32.890166 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:34.040430 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:34.040383 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:34.699311 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:34.699271 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:35.415547 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:35.415515 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:36.216471 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:36.216427 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:36.895675 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:36.895642 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:37.597122 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:37.597087 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:38.615428 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:38.615389 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:38.686309 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:38.686269 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:38.878651 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:38.878552 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:39.205057 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:39.205030 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13d31c-npcqm"] Sep 14 03:16:39.205529 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:39.205516 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51e4c384-415f-43ca-a05d-6e3d1f5ee5f3" containerName="payload-processing" Sep 14 03:16:39.205580 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:39.205531 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="51e4c384-415f-43ca-a05d-6e3d1f5ee5f3" containerName="payload-processing" Sep 14 03:16:39.205616 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:39.205609 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="51e4c384-415f-43ca-a05d-6e3d1f5ee5f3" containerName="payload-processing" Sep 14 03:16:39.210127 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:39.210111 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13d31c-npcqm" Sep 14 03:16:39.212199 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:39.212174 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4hf5p\"" Sep 14 03:16:39.212316 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:39.212174 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 03:16:39.216536 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:39.216511 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13d31c-npcqm"] Sep 14 03:16:39.298121 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:39.298090 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d1da26-aaa4-4bcf-9686-2e09d77fee05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-13d31c-npcqm\" (UID: \"b9d1da26-aaa4-4bcf-9686-2e09d77fee05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13d31c-npcqm" Sep 14 03:16:39.398802 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:39.398760 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d1da26-aaa4-4bcf-9686-2e09d77fee05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-13d31c-npcqm\" (UID: \"b9d1da26-aaa4-4bcf-9686-2e09d77fee05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13d31c-npcqm" Sep 14 03:16:39.398999 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:39.398761 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:39.399359 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:39.399341 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d1da26-aaa4-4bcf-9686-2e09d77fee05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-13d31c-npcqm\" (UID: \"b9d1da26-aaa4-4bcf-9686-2e09d77fee05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13d31c-npcqm" Sep 14 03:16:39.523766 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:39.523695 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13d31c-npcqm" Sep 14 03:16:39.801882 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:39.801859 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13d31c-npcqm"] Sep 14 03:16:39.804331 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:16:39.804308 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9d1da26_aaa4_4bcf_9686_2e09d77fee05.slice/crio-b40e440bf3fcc5e17ae4272a980dfb33d6f97ea05e3d41a3878f31bbc6c771fb WatchSource:0}: Error finding container b40e440bf3fcc5e17ae4272a980dfb33d6f97ea05e3d41a3878f31bbc6c771fb: Status 404 returned error can't find the container with id b40e440bf3fcc5e17ae4272a980dfb33d6f97ea05e3d41a3878f31bbc6c771fb Sep 14 03:16:40.236982 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:40.236949 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:40.596993 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:40.596941 2564 generic.go:356] "Generic (PLEG): container finished" podID="b9d1da26-aaa4-4bcf-9686-2e09d77fee05" containerID="6579390bfd48235fa70eba9fe2927f0e580719eccece818932337744e2612a9a" exitCode=0 Sep 14 03:16:40.597140 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:40.597022 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13d31c-npcqm" event={"ID":"b9d1da26-aaa4-4bcf-9686-2e09d77fee05","Type":"ContainerDied","Data":"6579390bfd48235fa70eba9fe2927f0e580719eccece818932337744e2612a9a"} Sep 14 03:16:40.597140 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:40.597090 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13d31c-npcqm" event={"ID":"b9d1da26-aaa4-4bcf-9686-2e09d77fee05","Type":"ContainerStarted","Data":"b40e440bf3fcc5e17ae4272a980dfb33d6f97ea05e3d41a3878f31bbc6c771fb"} Sep 14 03:16:40.893951 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:40.893910 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:41.734909 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:41.734865 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:41.743811 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:41.743788 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13d31c-npcqm" Sep 14 03:16:41.821243 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:41.821213 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d1da26-aaa4-4bcf-9686-2e09d77fee05-maas-api-service-ca\") pod \"b9d1da26-aaa4-4bcf-9686-2e09d77fee05\" (UID: \"b9d1da26-aaa4-4bcf-9686-2e09d77fee05\") " Sep 14 03:16:41.821670 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:41.821639 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d1da26-aaa4-4bcf-9686-2e09d77fee05-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b9d1da26-aaa4-4bcf-9686-2e09d77fee05" (UID: "b9d1da26-aaa4-4bcf-9686-2e09d77fee05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:16:41.923103 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:41.923071 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9d1da26-aaa4-4bcf-9686-2e09d77fee05-maas-api-service-ca\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:16:42.609771 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:42.609732 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13d31c-npcqm" event={"ID":"b9d1da26-aaa4-4bcf-9686-2e09d77fee05","Type":"ContainerDied","Data":"b40e440bf3fcc5e17ae4272a980dfb33d6f97ea05e3d41a3878f31bbc6c771fb"} Sep 14 03:16:42.609771 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:42.609754 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:42.610015 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:42.609789 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b40e440bf3fcc5e17ae4272a980dfb33d6f97ea05e3d41a3878f31bbc6c771fb" Sep 14 03:16:42.610015 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:42.609740 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13d31c-npcqm" Sep 14 03:16:43.440702 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:43.440659 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:44.329039 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:44.328998 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:44.864638 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:44.864606 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:45.547001 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:45.546967 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:49.998549 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:49.998505 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:54.939774 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:54.939740 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:54.966681 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:54.966649 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629"] Sep 14 03:16:54.966906 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:54.966882 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629" podUID="3c137e1f-0e2e-476d-8435-a0f4218569ed" containerName="payload-processing" containerID="cri-o://0a6675abf90fbfc85bdbc35fa30c8534a5fd28b4e2100bdf74325715d6929f45" gracePeriod=30 Sep 14 03:16:54.976133 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:54.976108 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb"] Sep 14 03:16:54.976412 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:54.976382 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb" podUID="014f38ab-a047-408b-9b4f-78638159e65b" containerName="payload-pre-processing" containerID="cri-o://0bf2049b553767efb65ed7bc3f063bd97559dbb5b3a0d019905671942ed1fd93" gracePeriod=30 Sep 14 03:16:55.042231 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.042205 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:55.144184 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.144068 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:55.183868 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.180857 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13d31c-npcqm"] Sep 14 03:16:55.184052 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.183899 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-13d31c-npcqm"] Sep 14 03:16:55.223101 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.223067 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629" Sep 14 03:16:55.243904 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.243667 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:55.260153 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.260130 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb" Sep 14 03:16:55.349755 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.349716 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fx8l\" (UniqueName: \"kubernetes.io/projected/014f38ab-a047-408b-9b4f-78638159e65b-kube-api-access-8fx8l\") pod \"014f38ab-a047-408b-9b4f-78638159e65b\" (UID: \"014f38ab-a047-408b-9b4f-78638159e65b\") " Sep 14 03:16:55.349934 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.349761 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c137e1f-0e2e-476d-8435-a0f4218569ed-plugins-config-volume\") pod \"3c137e1f-0e2e-476d-8435-a0f4218569ed\" (UID: \"3c137e1f-0e2e-476d-8435-a0f4218569ed\") " Sep 14 03:16:55.349934 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.349799 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/014f38ab-a047-408b-9b4f-78638159e65b-plugins-config-volume\") pod \"014f38ab-a047-408b-9b4f-78638159e65b\" (UID: \"014f38ab-a047-408b-9b4f-78638159e65b\") " Sep 14 03:16:55.349934 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.349828 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcd8l\" (UniqueName: \"kubernetes.io/projected/3c137e1f-0e2e-476d-8435-a0f4218569ed-kube-api-access-hcd8l\") pod \"3c137e1f-0e2e-476d-8435-a0f4218569ed\" (UID: \"3c137e1f-0e2e-476d-8435-a0f4218569ed\") " Sep 14 03:16:55.350232 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.350207 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c137e1f-0e2e-476d-8435-a0f4218569ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3c137e1f-0e2e-476d-8435-a0f4218569ed" (UID: "3c137e1f-0e2e-476d-8435-a0f4218569ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:16:55.350308 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.350208 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/014f38ab-a047-408b-9b4f-78638159e65b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "014f38ab-a047-408b-9b4f-78638159e65b" (UID: "014f38ab-a047-408b-9b4f-78638159e65b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:16:55.351936 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.351906 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014f38ab-a047-408b-9b4f-78638159e65b-kube-api-access-8fx8l" (OuterVolumeSpecName: "kube-api-access-8fx8l") pod "014f38ab-a047-408b-9b4f-78638159e65b" (UID: "014f38ab-a047-408b-9b4f-78638159e65b"). InnerVolumeSpecName "kube-api-access-8fx8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:16:55.352049 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.352001 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c137e1f-0e2e-476d-8435-a0f4218569ed-kube-api-access-hcd8l" (OuterVolumeSpecName: "kube-api-access-hcd8l") pod "3c137e1f-0e2e-476d-8435-a0f4218569ed" (UID: "3c137e1f-0e2e-476d-8435-a0f4218569ed"). InnerVolumeSpecName "kube-api-access-hcd8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:16:55.450985 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.450915 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fx8l\" (UniqueName: \"kubernetes.io/projected/014f38ab-a047-408b-9b4f-78638159e65b-kube-api-access-8fx8l\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:16:55.450985 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.450940 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c137e1f-0e2e-476d-8435-a0f4218569ed-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:16:55.450985 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.450951 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/014f38ab-a047-408b-9b4f-78638159e65b-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:16:55.450985 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.450959 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcd8l\" (UniqueName: \"kubernetes.io/projected/3c137e1f-0e2e-476d-8435-a0f4218569ed-kube-api-access-hcd8l\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:16:55.536233 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.536198 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:55.636337 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.636302 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:55.685062 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.685028 2564 generic.go:356] "Generic (PLEG): container finished" podID="3c137e1f-0e2e-476d-8435-a0f4218569ed" containerID="0a6675abf90fbfc85bdbc35fa30c8534a5fd28b4e2100bdf74325715d6929f45" exitCode=0 Sep 14 03:16:55.685220 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.685085 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629" Sep 14 03:16:55.685220 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.685109 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629" event={"ID":"3c137e1f-0e2e-476d-8435-a0f4218569ed","Type":"ContainerDied","Data":"0a6675abf90fbfc85bdbc35fa30c8534a5fd28b4e2100bdf74325715d6929f45"} Sep 14 03:16:55.685220 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.685179 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629" event={"ID":"3c137e1f-0e2e-476d-8435-a0f4218569ed","Type":"ContainerDied","Data":"91edf6eadac502f7744eaf753cec1296725fbdd4ef459def9cad3e5d1aada3c4"} Sep 14 03:16:55.685359 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.685212 2564 scope.go:117] "RemoveContainer" containerID="0a6675abf90fbfc85bdbc35fa30c8534a5fd28b4e2100bdf74325715d6929f45" Sep 14 03:16:55.687183 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.687076 2564 generic.go:356] "Generic (PLEG): container finished" podID="014f38ab-a047-408b-9b4f-78638159e65b" containerID="0bf2049b553767efb65ed7bc3f063bd97559dbb5b3a0d019905671942ed1fd93" exitCode=0 Sep 14 03:16:55.687183 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.687113 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb" event={"ID":"014f38ab-a047-408b-9b4f-78638159e65b","Type":"ContainerDied","Data":"0bf2049b553767efb65ed7bc3f063bd97559dbb5b3a0d019905671942ed1fd93"} Sep 14 03:16:55.687183 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.687120 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb" Sep 14 03:16:55.687183 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.687151 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb" event={"ID":"014f38ab-a047-408b-9b4f-78638159e65b","Type":"ContainerDied","Data":"98609046d5db15d335b9ecaf37d00e2b53aae6fc55a501ac949f08e1bc6e050b"} Sep 14 03:16:55.696200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.696185 2564 scope.go:117] "RemoveContainer" containerID="0a6675abf90fbfc85bdbc35fa30c8534a5fd28b4e2100bdf74325715d6929f45" Sep 14 03:16:55.696541 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:16:55.696513 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6675abf90fbfc85bdbc35fa30c8534a5fd28b4e2100bdf74325715d6929f45\": container with ID starting with 0a6675abf90fbfc85bdbc35fa30c8534a5fd28b4e2100bdf74325715d6929f45 not found: ID does not exist" containerID="0a6675abf90fbfc85bdbc35fa30c8534a5fd28b4e2100bdf74325715d6929f45" Sep 14 03:16:55.696646 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.696550 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6675abf90fbfc85bdbc35fa30c8534a5fd28b4e2100bdf74325715d6929f45"} err="failed to get container status \"0a6675abf90fbfc85bdbc35fa30c8534a5fd28b4e2100bdf74325715d6929f45\": rpc error: code = NotFound desc = could not find container \"0a6675abf90fbfc85bdbc35fa30c8534a5fd28b4e2100bdf74325715d6929f45\": container with ID starting with 0a6675abf90fbfc85bdbc35fa30c8534a5fd28b4e2100bdf74325715d6929f45 not found: ID does not exist" Sep 14 03:16:55.696646 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.696574 2564 scope.go:117] "RemoveContainer" containerID="0bf2049b553767efb65ed7bc3f063bd97559dbb5b3a0d019905671942ed1fd93" Sep 14 03:16:55.705281 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.705266 2564 scope.go:117] "RemoveContainer" containerID="0bf2049b553767efb65ed7bc3f063bd97559dbb5b3a0d019905671942ed1fd93" Sep 14 03:16:55.705549 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:16:55.705528 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bf2049b553767efb65ed7bc3f063bd97559dbb5b3a0d019905671942ed1fd93\": container with ID starting with 0bf2049b553767efb65ed7bc3f063bd97559dbb5b3a0d019905671942ed1fd93 not found: ID does not exist" containerID="0bf2049b553767efb65ed7bc3f063bd97559dbb5b3a0d019905671942ed1fd93" Sep 14 03:16:55.705622 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.705561 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bf2049b553767efb65ed7bc3f063bd97559dbb5b3a0d019905671942ed1fd93"} err="failed to get container status \"0bf2049b553767efb65ed7bc3f063bd97559dbb5b3a0d019905671942ed1fd93\": rpc error: code = NotFound desc = could not find container \"0bf2049b553767efb65ed7bc3f063bd97559dbb5b3a0d019905671942ed1fd93\": container with ID starting with 0bf2049b553767efb65ed7bc3f063bd97559dbb5b3a0d019905671942ed1fd93 not found: ID does not exist" Sep 14 03:16:55.718864 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.718842 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb"] Sep 14 03:16:55.724279 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.724261 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-13d31c-79d5fc8bdb-898cb"] Sep 14 03:16:55.734333 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.734310 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629"] Sep 14 03:16:55.739667 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:55.739648 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-13d31c-77d8cd56ff-8w629"] Sep 14 03:16:56.420912 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:56.420882 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:56.750884 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:56.750797 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="014f38ab-a047-408b-9b4f-78638159e65b" path="/var/lib/kubelet/pods/014f38ab-a047-408b-9b4f-78638159e65b/volumes" Sep 14 03:16:56.751225 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:56.751204 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c137e1f-0e2e-476d-8435-a0f4218569ed" path="/var/lib/kubelet/pods/3c137e1f-0e2e-476d-8435-a0f4218569ed/volumes" Sep 14 03:16:56.751583 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:56.751565 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9d1da26-aaa4-4bcf-9686-2e09d77fee05" path="/var/lib/kubelet/pods/b9d1da26-aaa4-4bcf-9686-2e09d77fee05/volumes" Sep 14 03:16:57.222240 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:57.222209 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:57.919452 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:57.919416 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:58.591171 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:58.591115 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:59.813358 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:59.813326 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:16:59.914150 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:16:59.914104 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:01.990099 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:01.990066 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:02.094816 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:02.094776 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:02.270066 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:02.269945 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:02.625048 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:02.624963 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6e87d0-rj2j7"] Sep 14 03:17:02.625408 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:02.625392 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9d1da26-aaa4-4bcf-9686-2e09d77fee05" containerName="revoke-keys" Sep 14 03:17:02.625499 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:02.625412 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9d1da26-aaa4-4bcf-9686-2e09d77fee05" containerName="revoke-keys" Sep 14 03:17:02.625499 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:02.625455 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c137e1f-0e2e-476d-8435-a0f4218569ed" containerName="payload-processing" Sep 14 03:17:02.625499 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:02.625464 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c137e1f-0e2e-476d-8435-a0f4218569ed" containerName="payload-processing" Sep 14 03:17:02.625499 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:02.625491 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="014f38ab-a047-408b-9b4f-78638159e65b" containerName="payload-pre-processing" Sep 14 03:17:02.625499 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:02.625500 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="014f38ab-a047-408b-9b4f-78638159e65b" containerName="payload-pre-processing" Sep 14 03:17:02.625695 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:02.625612 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9d1da26-aaa4-4bcf-9686-2e09d77fee05" containerName="revoke-keys" Sep 14 03:17:02.625695 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:02.625627 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="014f38ab-a047-408b-9b4f-78638159e65b" containerName="payload-pre-processing" Sep 14 03:17:02.625695 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:02.625637 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c137e1f-0e2e-476d-8435-a0f4218569ed" containerName="payload-processing" Sep 14 03:17:02.629084 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:02.629063 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6e87d0-rj2j7" Sep 14 03:17:02.631951 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:02.631929 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4hf5p\"" Sep 14 03:17:02.632267 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:02.632253 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 03:17:02.642430 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:02.642408 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6e87d0-rj2j7"] Sep 14 03:17:02.807385 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:02.807345 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:02.822963 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:02.822936 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03891b81-de6c-48b7-abff-e11d99d40b47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6e87d0-rj2j7\" (UID: \"03891b81-de6c-48b7-abff-e11d99d40b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6e87d0-rj2j7" Sep 14 03:17:02.923760 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:02.923733 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03891b81-de6c-48b7-abff-e11d99d40b47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6e87d0-rj2j7\" (UID: \"03891b81-de6c-48b7-abff-e11d99d40b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6e87d0-rj2j7" Sep 14 03:17:02.924310 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:02.924290 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03891b81-de6c-48b7-abff-e11d99d40b47-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-6e87d0-rj2j7\" (UID: \"03891b81-de6c-48b7-abff-e11d99d40b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6e87d0-rj2j7" Sep 14 03:17:02.939562 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:02.939529 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6e87d0-rj2j7" Sep 14 03:17:03.210303 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:03.210267 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6e87d0-rj2j7"] Sep 14 03:17:03.220143 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:17:03.220114 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03891b81_de6c_48b7_abff_e11d99d40b47.slice/crio-9e044c9e2ea4281c1e8fcd9b3f72400275ee41348f0a137c7e06bcf05b8fbd74 WatchSource:0}: Error finding container 9e044c9e2ea4281c1e8fcd9b3f72400275ee41348f0a137c7e06bcf05b8fbd74: Status 404 returned error can't find the container with id 9e044c9e2ea4281c1e8fcd9b3f72400275ee41348f0a137c7e06bcf05b8fbd74 Sep 14 03:17:03.733594 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:03.733510 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6e87d0-rj2j7" event={"ID":"03891b81-de6c-48b7-abff-e11d99d40b47","Type":"ContainerStarted","Data":"4d1fce75195c8a9a6c6e201be87505b29e1721a227261837505e3a8fad755411"} Sep 14 03:17:03.733594 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:03.733564 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6e87d0-rj2j7" event={"ID":"03891b81-de6c-48b7-abff-e11d99d40b47","Type":"ContainerStarted","Data":"9e044c9e2ea4281c1e8fcd9b3f72400275ee41348f0a137c7e06bcf05b8fbd74"} Sep 14 03:17:03.753081 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:03.753040 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6e87d0-rj2j7" podStartSLOduration=1.753026734 podStartE2EDuration="1.753026734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:17:03.751350073 +0000 UTC m=+1595.481009190" watchObservedRunningTime="2026-09-14 03:17:03.753026734 +0000 UTC m=+1595.482685847" Sep 14 03:17:03.761650 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:03.761624 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:04.461501 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:04.461456 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:04.747605 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:04.747520 2564 generic.go:356] "Generic (PLEG): container finished" podID="03891b81-de6c-48b7-abff-e11d99d40b47" containerID="4d1fce75195c8a9a6c6e201be87505b29e1721a227261837505e3a8fad755411" exitCode=0 Sep 14 03:17:04.750096 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:04.750069 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6e87d0-rj2j7" event={"ID":"03891b81-de6c-48b7-abff-e11d99d40b47","Type":"ContainerDied","Data":"4d1fce75195c8a9a6c6e201be87505b29e1721a227261837505e3a8fad755411"} Sep 14 03:17:05.344625 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:05.344588 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:05.890049 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:05.890018 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6e87d0-rj2j7" Sep 14 03:17:05.950799 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:05.950776 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03891b81-de6c-48b7-abff-e11d99d40b47-maas-api-service-ca\") pod \"03891b81-de6c-48b7-abff-e11d99d40b47\" (UID: \"03891b81-de6c-48b7-abff-e11d99d40b47\") " Sep 14 03:17:05.951090 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:05.951068 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03891b81-de6c-48b7-abff-e11d99d40b47-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "03891b81-de6c-48b7-abff-e11d99d40b47" (UID: "03891b81-de6c-48b7-abff-e11d99d40b47"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:17:06.051925 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:06.051839 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03891b81-de6c-48b7-abff-e11d99d40b47-maas-api-service-ca\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:17:06.108629 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:06.108588 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:06.212101 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:06.212068 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9_3fada7bb-85e2-4680-ab7b-e7dde698eca3/payload-processing/0.log" Sep 14 03:17:06.760869 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:06.760847 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6e87d0-rj2j7" Sep 14 03:17:06.761015 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:06.760891 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6e87d0-rj2j7" event={"ID":"03891b81-de6c-48b7-abff-e11d99d40b47","Type":"ContainerDied","Data":"9e044c9e2ea4281c1e8fcd9b3f72400275ee41348f0a137c7e06bcf05b8fbd74"} Sep 14 03:17:06.761015 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:06.760951 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e044c9e2ea4281c1e8fcd9b3f72400275ee41348f0a137c7e06bcf05b8fbd74" Sep 14 03:17:06.808553 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:06.808520 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:06.948987 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:06.948954 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:07.225544 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.225511 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf"] Sep 14 03:17:07.225960 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.225946 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03891b81-de6c-48b7-abff-e11d99d40b47" containerName="revoke-keys" Sep 14 03:17:07.225960 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.225961 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="03891b81-de6c-48b7-abff-e11d99d40b47" containerName="revoke-keys" Sep 14 03:17:07.226105 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.226027 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="03891b81-de6c-48b7-abff-e11d99d40b47" containerName="revoke-keys" Sep 14 03:17:07.229077 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.229062 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.231358 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.231340 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-a35f4b-openshift-default-dockercfg-tvwpl\"" Sep 14 03:17:07.240423 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.240399 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf"] Sep 14 03:17:07.260434 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.260398 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:07.263991 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.263964 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6ff6da-345b-4e30-b748-723c74edaca6-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.264115 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.264008 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thzk4\" (UniqueName: \"kubernetes.io/projected/cc6ff6da-345b-4e30-b748-723c74edaca6-kube-api-access-thzk4\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.264115 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.264096 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-credential-socket\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.264258 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.264124 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-workload-socket\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.264258 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.264154 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-token\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.264258 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.264221 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-workload-certs\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.264544 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.264522 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-podinfo\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.264636 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.264591 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-data\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.264636 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.264624 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-envoy\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.365240 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.365204 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6ff6da-345b-4e30-b748-723c74edaca6-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.365585 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.365256 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thzk4\" (UniqueName: \"kubernetes.io/projected/cc6ff6da-345b-4e30-b748-723c74edaca6-kube-api-access-thzk4\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.365585 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.365307 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-credential-socket\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.365585 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.365335 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-workload-socket\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.365585 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.365363 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-token\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.365585 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.365423 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-workload-certs\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.365585 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.365457 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-podinfo\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.365585 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.365507 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-data\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.365988 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.365756 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-credential-socket\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.365988 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.365864 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-workload-certs\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.365988 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.365962 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-workload-socket\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.366147 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.366040 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-envoy\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.366147 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.366053 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-data\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.366147 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.366084 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6ff6da-345b-4e30-b748-723c74edaca6-istiod-ca-cert\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.368194 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.368173 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-envoy\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.368290 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.368216 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-podinfo\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.382715 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.382653 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-token\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.383239 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.383222 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thzk4\" (UniqueName: \"kubernetes.io/projected/cc6ff6da-345b-4e30-b748-723c74edaca6-kube-api-access-thzk4\") pod \"e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.462643 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.462604 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:07.534407 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.534372 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:07.541206 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.541181 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:07.829879 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.829846 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf"] Sep 14 03:17:07.833612 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:17:07.833583 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6ff6da_345b_4e30_b748_723c74edaca6.slice/crio-2abc689f6af3eb0c0bbe39d4405e9a20411e1e4bb3440d3bec94b014335e45da WatchSource:0}: Error finding container 2abc689f6af3eb0c0bbe39d4405e9a20411e1e4bb3440d3bec94b014335e45da: Status 404 returned error can't find the container with id 2abc689f6af3eb0c0bbe39d4405e9a20411e1e4bb3440d3bec94b014335e45da Sep 14 03:17:07.835947 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.835912 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:17:07.836029 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.835980 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:17:07.836029 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:07.836011 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:17:08.344364 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:08.344331 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:08.772604 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:08.772570 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" event={"ID":"cc6ff6da-345b-4e30-b748-723c74edaca6","Type":"ContainerStarted","Data":"907314fbb654eac142998d522d20c933820a3118ba4b04b41605b23ea6176fc8"} Sep 14 03:17:08.772781 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:08.772619 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" event={"ID":"cc6ff6da-345b-4e30-b748-723c74edaca6","Type":"ContainerStarted","Data":"2abc689f6af3eb0c0bbe39d4405e9a20411e1e4bb3440d3bec94b014335e45da"} Sep 14 03:17:08.810867 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:08.810814 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" podStartSLOduration=1.810796364 podStartE2EDuration="1.810796364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:17:08.808890134 +0000 UTC m=+1600.538549251" watchObservedRunningTime="2026-09-14 03:17:08.810796364 +0000 UTC m=+1600.540455481" Sep 14 03:17:09.541313 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:09.541272 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:09.546675 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:09.546648 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:09.778659 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:09.778626 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:09.779746 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:09.779718 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:10.509533 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:10.509466 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:10.789504 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:10.789389 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:10.880277 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:10.880234 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:11.472706 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:11.472678 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:12.356322 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:12.356289 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:13.256119 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:13.256082 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:14.144524 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:14.144461 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:14.753611 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:14.753575 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:15.637349 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:15.637307 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:15.736286 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:15.736246 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:15.859958 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:15.859894 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:16.127577 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.127525 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:16.229412 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.229373 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:16.314411 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.314382 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2"] Sep 14 03:17:16.318294 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.318278 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2" Sep 14 03:17:16.320629 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.320612 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-a35f4b\"" Sep 14 03:17:16.320724 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.320656 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-a35f4b-dockercfg-svgjk\"" Sep 14 03:17:16.331710 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.331685 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2"] Sep 14 03:17:16.351088 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.351062 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab284e8-e85c-49d2-b959-1fc69c076cb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2\" (UID: \"6ab284e8-e85c-49d2-b959-1fc69c076cb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2" Sep 14 03:17:16.351251 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.351207 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncx6f\" (UniqueName: \"kubernetes.io/projected/6ab284e8-e85c-49d2-b959-1fc69c076cb9-kube-api-access-ncx6f\") pod \"payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2\" (UID: \"6ab284e8-e85c-49d2-b959-1fc69c076cb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2" Sep 14 03:17:16.452067 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.452037 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncx6f\" (UniqueName: \"kubernetes.io/projected/6ab284e8-e85c-49d2-b959-1fc69c076cb9-kube-api-access-ncx6f\") pod \"payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2\" (UID: \"6ab284e8-e85c-49d2-b959-1fc69c076cb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2" Sep 14 03:17:16.452269 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.452098 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab284e8-e85c-49d2-b959-1fc69c076cb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2\" (UID: \"6ab284e8-e85c-49d2-b959-1fc69c076cb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2" Sep 14 03:17:16.453057 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.453028 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab284e8-e85c-49d2-b959-1fc69c076cb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2\" (UID: \"6ab284e8-e85c-49d2-b959-1fc69c076cb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2" Sep 14 03:17:16.459048 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.459008 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:16.460918 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.460896 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncx6f\" (UniqueName: \"kubernetes.io/projected/6ab284e8-e85c-49d2-b959-1fc69c076cb9-kube-api-access-ncx6f\") pod \"payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2\" (UID: \"6ab284e8-e85c-49d2-b959-1fc69c076cb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2" Sep 14 03:17:16.491627 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.491594 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm"] Sep 14 03:17:16.491873 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.491846 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm" podUID="6125ac4f-31f5-4a0c-a000-5ef315c3f11c" containerName="payload-processing" containerID="cri-o://8b037616e0e554a6f44c5f136a69cb7a5f9ccc871d4c5f5d0c4dd9d8628cd0f7" gracePeriod=30 Sep 14 03:17:16.500613 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.500590 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m"] Sep 14 03:17:16.500810 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.500790 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m" podUID="01ddce15-f22b-4942-a53b-b30b415315ff" containerName="payload-pre-processing" containerID="cri-o://4250d55f220c5ac23658c9fd3758f58b20ccec37cbda905768dc62791cfb0462" gracePeriod=30 Sep 14 03:17:16.540937 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.540906 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:16.614671 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.614638 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:16.629433 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.629402 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2" Sep 14 03:17:16.696650 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.694045 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6e87d0-rj2j7"] Sep 14 03:17:16.699130 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.699103 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-6e87d0-rj2j7"] Sep 14 03:17:16.699704 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.699671 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:16.754947 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.754881 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03891b81-de6c-48b7-abff-e11d99d40b47" path="/var/lib/kubelet/pods/03891b81-de6c-48b7-abff-e11d99d40b47/volumes" Sep 14 03:17:16.757898 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.757877 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm" Sep 14 03:17:16.783462 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.783061 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:16.796302 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.796284 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m" Sep 14 03:17:16.822515 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.822468 2564 generic.go:356] "Generic (PLEG): container finished" podID="01ddce15-f22b-4942-a53b-b30b415315ff" containerID="4250d55f220c5ac23658c9fd3758f58b20ccec37cbda905768dc62791cfb0462" exitCode=0 Sep 14 03:17:16.822674 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.822550 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m" Sep 14 03:17:16.822674 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.822594 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m" event={"ID":"01ddce15-f22b-4942-a53b-b30b415315ff","Type":"ContainerDied","Data":"4250d55f220c5ac23658c9fd3758f58b20ccec37cbda905768dc62791cfb0462"} Sep 14 03:17:16.822674 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.822656 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m" event={"ID":"01ddce15-f22b-4942-a53b-b30b415315ff","Type":"ContainerDied","Data":"5a621dede5ebf3b446af9bc73c2da8b12ddb2a74fd8571bf7b97a9b98463d2a9"} Sep 14 03:17:16.822851 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.822686 2564 scope.go:117] "RemoveContainer" containerID="4250d55f220c5ac23658c9fd3758f58b20ccec37cbda905768dc62791cfb0462" Sep 14 03:17:16.824188 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.824164 2564 generic.go:356] "Generic (PLEG): container finished" podID="6125ac4f-31f5-4a0c-a000-5ef315c3f11c" containerID="8b037616e0e554a6f44c5f136a69cb7a5f9ccc871d4c5f5d0c4dd9d8628cd0f7" exitCode=0 Sep 14 03:17:16.824311 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.824219 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm" Sep 14 03:17:16.824311 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.824245 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm" event={"ID":"6125ac4f-31f5-4a0c-a000-5ef315c3f11c","Type":"ContainerDied","Data":"8b037616e0e554a6f44c5f136a69cb7a5f9ccc871d4c5f5d0c4dd9d8628cd0f7"} Sep 14 03:17:16.824311 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.824293 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm" event={"ID":"6125ac4f-31f5-4a0c-a000-5ef315c3f11c","Type":"ContainerDied","Data":"9f071e7b36d5a0da6a08497c0dd56fd0f6b935e79c4ac3041851d299660093b1"} Sep 14 03:17:16.832256 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.832237 2564 scope.go:117] "RemoveContainer" containerID="4250d55f220c5ac23658c9fd3758f58b20ccec37cbda905768dc62791cfb0462" Sep 14 03:17:16.832588 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:17:16.832558 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4250d55f220c5ac23658c9fd3758f58b20ccec37cbda905768dc62791cfb0462\": container with ID starting with 4250d55f220c5ac23658c9fd3758f58b20ccec37cbda905768dc62791cfb0462 not found: ID does not exist" containerID="4250d55f220c5ac23658c9fd3758f58b20ccec37cbda905768dc62791cfb0462" Sep 14 03:17:16.832677 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.832600 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4250d55f220c5ac23658c9fd3758f58b20ccec37cbda905768dc62791cfb0462"} err="failed to get container status \"4250d55f220c5ac23658c9fd3758f58b20ccec37cbda905768dc62791cfb0462\": rpc error: code = NotFound desc = could not find container \"4250d55f220c5ac23658c9fd3758f58b20ccec37cbda905768dc62791cfb0462\": container with ID starting with 4250d55f220c5ac23658c9fd3758f58b20ccec37cbda905768dc62791cfb0462 not found: ID does not exist" Sep 14 03:17:16.832677 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.832622 2564 scope.go:117] "RemoveContainer" containerID="8b037616e0e554a6f44c5f136a69cb7a5f9ccc871d4c5f5d0c4dd9d8628cd0f7" Sep 14 03:17:16.842743 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.842724 2564 scope.go:117] "RemoveContainer" containerID="8b037616e0e554a6f44c5f136a69cb7a5f9ccc871d4c5f5d0c4dd9d8628cd0f7" Sep 14 03:17:16.843096 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:17:16.843076 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b037616e0e554a6f44c5f136a69cb7a5f9ccc871d4c5f5d0c4dd9d8628cd0f7\": container with ID starting with 8b037616e0e554a6f44c5f136a69cb7a5f9ccc871d4c5f5d0c4dd9d8628cd0f7 not found: ID does not exist" containerID="8b037616e0e554a6f44c5f136a69cb7a5f9ccc871d4c5f5d0c4dd9d8628cd0f7" Sep 14 03:17:16.843182 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.843106 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b037616e0e554a6f44c5f136a69cb7a5f9ccc871d4c5f5d0c4dd9d8628cd0f7"} err="failed to get container status \"8b037616e0e554a6f44c5f136a69cb7a5f9ccc871d4c5f5d0c4dd9d8628cd0f7\": rpc error: code = NotFound desc = could not find container \"8b037616e0e554a6f44c5f136a69cb7a5f9ccc871d4c5f5d0c4dd9d8628cd0f7\": container with ID starting with 8b037616e0e554a6f44c5f136a69cb7a5f9ccc871d4c5f5d0c4dd9d8628cd0f7 not found: ID does not exist" Sep 14 03:17:16.855613 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.855586 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2j2s\" (UniqueName: \"kubernetes.io/projected/01ddce15-f22b-4942-a53b-b30b415315ff-kube-api-access-j2j2s\") pod \"01ddce15-f22b-4942-a53b-b30b415315ff\" (UID: \"01ddce15-f22b-4942-a53b-b30b415315ff\") " Sep 14 03:17:16.855763 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.855631 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfn7j\" (UniqueName: \"kubernetes.io/projected/6125ac4f-31f5-4a0c-a000-5ef315c3f11c-kube-api-access-sfn7j\") pod \"6125ac4f-31f5-4a0c-a000-5ef315c3f11c\" (UID: \"6125ac4f-31f5-4a0c-a000-5ef315c3f11c\") " Sep 14 03:17:16.855763 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.855686 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6125ac4f-31f5-4a0c-a000-5ef315c3f11c-plugins-config-volume\") pod \"6125ac4f-31f5-4a0c-a000-5ef315c3f11c\" (UID: \"6125ac4f-31f5-4a0c-a000-5ef315c3f11c\") " Sep 14 03:17:16.855763 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.855702 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01ddce15-f22b-4942-a53b-b30b415315ff-plugins-config-volume\") pod \"01ddce15-f22b-4942-a53b-b30b415315ff\" (UID: \"01ddce15-f22b-4942-a53b-b30b415315ff\") " Sep 14 03:17:16.856345 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.856208 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ddce15-f22b-4942-a53b-b30b415315ff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01ddce15-f22b-4942-a53b-b30b415315ff" (UID: "01ddce15-f22b-4942-a53b-b30b415315ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:17:16.856345 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.856286 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6125ac4f-31f5-4a0c-a000-5ef315c3f11c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6125ac4f-31f5-4a0c-a000-5ef315c3f11c" (UID: "6125ac4f-31f5-4a0c-a000-5ef315c3f11c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:17:16.858245 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.858224 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ddce15-f22b-4942-a53b-b30b415315ff-kube-api-access-j2j2s" (OuterVolumeSpecName: "kube-api-access-j2j2s") pod "01ddce15-f22b-4942-a53b-b30b415315ff" (UID: "01ddce15-f22b-4942-a53b-b30b415315ff"). InnerVolumeSpecName "kube-api-access-j2j2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:17:16.858542 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.858456 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6125ac4f-31f5-4a0c-a000-5ef315c3f11c-kube-api-access-sfn7j" (OuterVolumeSpecName: "kube-api-access-sfn7j") pod "6125ac4f-31f5-4a0c-a000-5ef315c3f11c" (UID: "6125ac4f-31f5-4a0c-a000-5ef315c3f11c"). InnerVolumeSpecName "kube-api-access-sfn7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:17:16.923349 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.923318 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2"] Sep 14 03:17:16.923675 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:17:16.923654 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab284e8_e85c_49d2_b959_1fc69c076cb9.slice/crio-6c5acf73bdd44d9b37f73f79e8682e14496609a892f9d69b44bf4e2707014104 WatchSource:0}: Error finding container 6c5acf73bdd44d9b37f73f79e8682e14496609a892f9d69b44bf4e2707014104: Status 404 returned error can't find the container with id 6c5acf73bdd44d9b37f73f79e8682e14496609a892f9d69b44bf4e2707014104 Sep 14 03:17:16.957363 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.957336 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2j2s\" (UniqueName: \"kubernetes.io/projected/01ddce15-f22b-4942-a53b-b30b415315ff-kube-api-access-j2j2s\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:17:16.957363 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.957362 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfn7j\" (UniqueName: \"kubernetes.io/projected/6125ac4f-31f5-4a0c-a000-5ef315c3f11c-kube-api-access-sfn7j\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:17:16.957509 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.957373 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6125ac4f-31f5-4a0c-a000-5ef315c3f11c-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:17:16.957509 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:16.957381 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01ddce15-f22b-4942-a53b-b30b415315ff-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:17:17.137543 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:17.137516 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:17.174260 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:17.174227 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m"] Sep 14 03:17:17.185068 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:17.185042 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-6e87d0-7f7bfc4cf8-hk69m"] Sep 14 03:17:17.210300 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:17.210276 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm"] Sep 14 03:17:17.234370 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:17.234315 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-6e87d0-6ddb98b6f5-kgqzm"] Sep 14 03:17:17.506955 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:17.506866 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:17.834343 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:17.834240 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2" event={"ID":"6ab284e8-e85c-49d2-b959-1fc69c076cb9","Type":"ContainerStarted","Data":"ed1407efe744cf78de1224413501cab2410eb9c52235b235bf286a04ecda123e"} Sep 14 03:17:17.834343 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:17.834301 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2" event={"ID":"6ab284e8-e85c-49d2-b959-1fc69c076cb9","Type":"ContainerStarted","Data":"6c5acf73bdd44d9b37f73f79e8682e14496609a892f9d69b44bf4e2707014104"} Sep 14 03:17:17.834343 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:17.834320 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2" Sep 14 03:17:17.854428 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:17.854376 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2" podStartSLOduration=1.854358393 podStartE2EDuration="1.854358393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:17:17.852099889 +0000 UTC m=+1609.581759030" watchObservedRunningTime="2026-09-14 03:17:17.854358393 +0000 UTC m=+1609.584017555" Sep 14 03:17:18.216039 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:18.216005 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:18.751798 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:18.751760 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01ddce15-f22b-4942-a53b-b30b415315ff" path="/var/lib/kubelet/pods/01ddce15-f22b-4942-a53b-b30b415315ff/volumes" Sep 14 03:17:18.752177 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:18.752164 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6125ac4f-31f5-4a0c-a000-5ef315c3f11c" path="/var/lib/kubelet/pods/6125ac4f-31f5-4a0c-a000-5ef315c3f11c/volumes" Sep 14 03:17:19.163550 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:19.163513 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:19.852660 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:19.852608 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:20.608382 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:20.608344 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:21.342299 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:21.342263 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:22.230591 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:22.230541 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:23.568938 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:23.568909 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:23.659313 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:23.659279 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:23.840141 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:23.840029 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:24.171238 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:24.171202 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f77a9-q6w8x"] Sep 14 03:17:24.171639 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:24.171625 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01ddce15-f22b-4942-a53b-b30b415315ff" containerName="payload-pre-processing" Sep 14 03:17:24.171639 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:24.171639 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="01ddce15-f22b-4942-a53b-b30b415315ff" containerName="payload-pre-processing" Sep 14 03:17:24.171734 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:24.171660 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6125ac4f-31f5-4a0c-a000-5ef315c3f11c" containerName="payload-processing" Sep 14 03:17:24.171734 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:24.171666 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6125ac4f-31f5-4a0c-a000-5ef315c3f11c" containerName="payload-processing" Sep 14 03:17:24.171803 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:24.171746 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="01ddce15-f22b-4942-a53b-b30b415315ff" containerName="payload-pre-processing" Sep 14 03:17:24.171803 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:24.171759 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6125ac4f-31f5-4a0c-a000-5ef315c3f11c" containerName="payload-processing" Sep 14 03:17:24.176512 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:24.176466 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f77a9-q6w8x" Sep 14 03:17:24.179369 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:24.179324 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 03:17:24.179621 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:24.179601 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4hf5p\"" Sep 14 03:17:24.185460 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:24.185428 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f77a9-q6w8x"] Sep 14 03:17:24.232607 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:24.232580 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe754cc5-a71e-4088-a9d6-618fbb31d97f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9f77a9-q6w8x\" (UID: \"fe754cc5-a71e-4088-a9d6-618fbb31d97f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f77a9-q6w8x" Sep 14 03:17:24.333706 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:24.333670 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe754cc5-a71e-4088-a9d6-618fbb31d97f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9f77a9-q6w8x\" (UID: \"fe754cc5-a71e-4088-a9d6-618fbb31d97f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f77a9-q6w8x" Sep 14 03:17:24.334539 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:24.334509 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe754cc5-a71e-4088-a9d6-618fbb31d97f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-9f77a9-q6w8x\" (UID: \"fe754cc5-a71e-4088-a9d6-618fbb31d97f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f77a9-q6w8x" Sep 14 03:17:24.373915 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:24.373878 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:24.498676 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:24.498581 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f77a9-q6w8x" Sep 14 03:17:24.771277 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:24.771249 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f77a9-q6w8x"] Sep 14 03:17:24.772277 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:17:24.772256 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe754cc5_a71e_4088_a9d6_618fbb31d97f.slice/crio-e5c00f714bc9748b03c57f7f2365159bd4dd26fc6885550c68812e36a25e19da WatchSource:0}: Error finding container e5c00f714bc9748b03c57f7f2365159bd4dd26fc6885550c68812e36a25e19da: Status 404 returned error can't find the container with id e5c00f714bc9748b03c57f7f2365159bd4dd26fc6885550c68812e36a25e19da Sep 14 03:17:24.873251 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:24.873226 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f77a9-q6w8x" event={"ID":"fe754cc5-a71e-4088-a9d6-618fbb31d97f","Type":"ContainerStarted","Data":"e5c00f714bc9748b03c57f7f2365159bd4dd26fc6885550c68812e36a25e19da"} Sep 14 03:17:25.104415 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:25.104380 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:25.883767 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:25.883728 2564 generic.go:356] "Generic (PLEG): container finished" podID="fe754cc5-a71e-4088-a9d6-618fbb31d97f" containerID="dce11b70c0cf78c2522fb5c27d0e6b2d954b0a04a320b58b9adc5dabaa2561e2" exitCode=0 Sep 14 03:17:25.884357 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:25.883804 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f77a9-q6w8x" event={"ID":"fe754cc5-a71e-4088-a9d6-618fbb31d97f","Type":"ContainerDied","Data":"dce11b70c0cf78c2522fb5c27d0e6b2d954b0a04a320b58b9adc5dabaa2561e2"} Sep 14 03:17:25.915075 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:25.915042 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:26.609503 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:26.609421 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:27.058519 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:27.058494 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f77a9-q6w8x" Sep 14 03:17:27.163110 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:27.163036 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe754cc5-a71e-4088-a9d6-618fbb31d97f-maas-api-service-ca\") pod \"fe754cc5-a71e-4088-a9d6-618fbb31d97f\" (UID: \"fe754cc5-a71e-4088-a9d6-618fbb31d97f\") " Sep 14 03:17:27.163355 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:27.163334 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe754cc5-a71e-4088-a9d6-618fbb31d97f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fe754cc5-a71e-4088-a9d6-618fbb31d97f" (UID: "fe754cc5-a71e-4088-a9d6-618fbb31d97f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:17:27.264321 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:27.264286 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe754cc5-a71e-4088-a9d6-618fbb31d97f-maas-api-service-ca\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:17:27.553856 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:27.553765 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:27.899266 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:27.899237 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f77a9-q6w8x" Sep 14 03:17:27.899416 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:27.899236 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f77a9-q6w8x" event={"ID":"fe754cc5-a71e-4088-a9d6-618fbb31d97f","Type":"ContainerDied","Data":"e5c00f714bc9748b03c57f7f2365159bd4dd26fc6885550c68812e36a25e19da"} Sep 14 03:17:27.899416 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:27.899361 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5c00f714bc9748b03c57f7f2365159bd4dd26fc6885550c68812e36a25e19da" Sep 14 03:17:28.282725 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:28.282627 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:28.817328 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:28.817296 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:28.841948 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:28.841916 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2" Sep 14 03:17:29.497254 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:29.497221 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:30.235302 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:30.235262 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:30.492668 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:30.492577 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:30.920216 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:30.920178 2564 generic.go:356] "Generic (PLEG): container finished" podID="cc6ff6da-345b-4e30-b748-723c74edaca6" containerID="907314fbb654eac142998d522d20c933820a3118ba4b04b41605b23ea6176fc8" exitCode=0 Sep 14 03:17:30.920655 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:30.920236 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" event={"ID":"cc6ff6da-345b-4e30-b748-723c74edaca6","Type":"ContainerDied","Data":"907314fbb654eac142998d522d20c933820a3118ba4b04b41605b23ea6176fc8"} Sep 14 03:17:30.920703 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:30.920657 2564 scope.go:117] "RemoveContainer" containerID="907314fbb654eac142998d522d20c933820a3118ba4b04b41605b23ea6176fc8" Sep 14 03:17:30.921941 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:30.921908 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:17:30.922023 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:30.921982 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:17:30.922023 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:30.922014 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 14 03:17:30.976657 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:30.976620 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:31.156789 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:31.156757 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35f4b-tzlzj"] Sep 14 03:17:31.157194 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:31.157179 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe754cc5-a71e-4088-a9d6-618fbb31d97f" containerName="revoke-keys" Sep 14 03:17:31.157194 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:31.157192 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe754cc5-a71e-4088-a9d6-618fbb31d97f" containerName="revoke-keys" Sep 14 03:17:31.157359 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:31.157266 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe754cc5-a71e-4088-a9d6-618fbb31d97f" containerName="revoke-keys" Sep 14 03:17:31.160392 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:31.160374 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35f4b-tzlzj" Sep 14 03:17:31.163105 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:31.163084 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4hf5p\"" Sep 14 03:17:31.163655 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:31.163637 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 03:17:31.169651 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:31.169588 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35f4b-tzlzj"] Sep 14 03:17:31.310768 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:31.310735 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6fc5e8d-ec6c-47b7-ae8e-22f7f7b94216-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a35f4b-tzlzj\" (UID: \"c6fc5e8d-ec6c-47b7-ae8e-22f7f7b94216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35f4b-tzlzj" Sep 14 03:17:31.412070 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:31.411985 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6fc5e8d-ec6c-47b7-ae8e-22f7f7b94216-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a35f4b-tzlzj\" (UID: \"c6fc5e8d-ec6c-47b7-ae8e-22f7f7b94216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35f4b-tzlzj" Sep 14 03:17:31.412642 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:31.412618 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6fc5e8d-ec6c-47b7-ae8e-22f7f7b94216-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-a35f4b-tzlzj\" (UID: \"c6fc5e8d-ec6c-47b7-ae8e-22f7f7b94216\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35f4b-tzlzj" Sep 14 03:17:31.474019 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:31.473989 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35f4b-tzlzj" Sep 14 03:17:31.752462 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:31.752436 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35f4b-tzlzj"] Sep 14 03:17:31.753989 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:17:31.753955 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6fc5e8d_ec6c_47b7_ae8e_22f7f7b94216.slice/crio-4d26ee08f80c0514d0e80af182780e7b5aff9763748ed10717dc958054e6fe09 WatchSource:0}: Error finding container 4d26ee08f80c0514d0e80af182780e7b5aff9763748ed10717dc958054e6fe09: Status 404 returned error can't find the container with id 4d26ee08f80c0514d0e80af182780e7b5aff9763748ed10717dc958054e6fe09 Sep 14 03:17:31.927193 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:31.927159 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35f4b-tzlzj" event={"ID":"c6fc5e8d-ec6c-47b7-ae8e-22f7f7b94216","Type":"ContainerStarted","Data":"4d26ee08f80c0514d0e80af182780e7b5aff9763748ed10717dc958054e6fe09"} Sep 14 03:17:31.929168 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:31.929146 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" event={"ID":"cc6ff6da-345b-4e30-b748-723c74edaca6","Type":"ContainerStarted","Data":"99e68ccb63708c048b0c8bdba3f635d5c067636d5fa1ddc81d3832faf18dd2b0"} Sep 14 03:17:31.933150 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:31.933127 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:32.542033 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:32.542000 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:32.542283 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:32.542259 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 14 03:17:32.542344 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:32.542308 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" podUID="cc6ff6da-345b-4e30-b748-723c74edaca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 14 03:17:32.706828 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:32.706731 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:32.936467 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:32.936434 2564 generic.go:356] "Generic (PLEG): container finished" podID="c6fc5e8d-ec6c-47b7-ae8e-22f7f7b94216" containerID="92e243c1c370e0ea8c498086d67e4a2b8d4ce733991e6b52305541fee36bbe85" exitCode=0 Sep 14 03:17:32.936896 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:32.936524 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35f4b-tzlzj" event={"ID":"c6fc5e8d-ec6c-47b7-ae8e-22f7f7b94216","Type":"ContainerDied","Data":"92e243c1c370e0ea8c498086d67e4a2b8d4ce733991e6b52305541fee36bbe85"} Sep 14 03:17:33.457738 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:33.457695 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:33.542224 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:33.542191 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 14 03:17:33.542393 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:33.542244 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" podUID="cc6ff6da-345b-4e30-b748-723c74edaca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 14 03:17:34.144010 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:34.143984 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35f4b-tzlzj" Sep 14 03:17:34.245912 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:34.245827 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6fc5e8d-ec6c-47b7-ae8e-22f7f7b94216-maas-api-service-ca\") pod \"c6fc5e8d-ec6c-47b7-ae8e-22f7f7b94216\" (UID: \"c6fc5e8d-ec6c-47b7-ae8e-22f7f7b94216\") " Sep 14 03:17:34.246161 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:34.246136 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6fc5e8d-ec6c-47b7-ae8e-22f7f7b94216-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c6fc5e8d-ec6c-47b7-ae8e-22f7f7b94216" (UID: "c6fc5e8d-ec6c-47b7-ae8e-22f7f7b94216"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:17:34.347298 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:34.347264 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6fc5e8d-ec6c-47b7-ae8e-22f7f7b94216-maas-api-service-ca\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:17:34.435940 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:34.435903 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:34.542670 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:34.542583 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 14 03:17:34.542670 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:34.542639 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" podUID="cc6ff6da-345b-4e30-b748-723c74edaca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 14 03:17:34.949455 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:34.949424 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35f4b-tzlzj" Sep 14 03:17:34.949455 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:34.949439 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35f4b-tzlzj" event={"ID":"c6fc5e8d-ec6c-47b7-ae8e-22f7f7b94216","Type":"ContainerDied","Data":"4d26ee08f80c0514d0e80af182780e7b5aff9763748ed10717dc958054e6fe09"} Sep 14 03:17:34.949686 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:34.949500 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d26ee08f80c0514d0e80af182780e7b5aff9763748ed10717dc958054e6fe09" Sep 14 03:17:35.112093 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:35.112056 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:35.542305 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:35.542271 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 14 03:17:35.542717 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:35.542333 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" podUID="cc6ff6da-345b-4e30-b748-723c74edaca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 14 03:17:35.831241 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:35.831145 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:36.350994 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:36.350964 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:36.542307 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:36.542268 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 14 03:17:36.542501 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:36.542326 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" podUID="cc6ff6da-345b-4e30-b748-723c74edaca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 14 03:17:36.705530 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:36.705468 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:37.368935 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.368882 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f77a9-29822595-fqczj"] Sep 14 03:17:37.373395 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.373360 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:37.383503 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.375239 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-9f77a9-29822595-fqczj"] Sep 14 03:17:37.410885 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.410852 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n"] Sep 14 03:17:37.411126 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.411098 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n" podUID="202a2300-292b-4bb2-87e0-ce7dfe7526e4" containerName="payload-pre-processing" containerID="cri-o://1581f5e4af418e878ac05e349248678cda73cc5edd922e5a0028f82233995709" gracePeriod=30 Sep 14 03:17:37.484510 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.479945 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:37.541899 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.541872 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Sep 14 03:17:37.542015 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.541930 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" podUID="cc6ff6da-345b-4e30-b748-723c74edaca6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Sep 14 03:17:37.583504 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.580999 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:37.612077 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.612046 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f77a9-q6w8x"] Sep 14 03:17:37.615612 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.615589 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-9f77a9-q6w8x"] Sep 14 03:17:37.696243 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.695587 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:37.702540 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.702514 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n" Sep 14 03:17:37.819274 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.819241 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:37.891659 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.891621 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pv5v\" (UniqueName: \"kubernetes.io/projected/202a2300-292b-4bb2-87e0-ce7dfe7526e4-kube-api-access-5pv5v\") pod \"202a2300-292b-4bb2-87e0-ce7dfe7526e4\" (UID: \"202a2300-292b-4bb2-87e0-ce7dfe7526e4\") " Sep 14 03:17:37.891851 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.891749 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202a2300-292b-4bb2-87e0-ce7dfe7526e4-plugins-config-volume\") pod \"202a2300-292b-4bb2-87e0-ce7dfe7526e4\" (UID: \"202a2300-292b-4bb2-87e0-ce7dfe7526e4\") " Sep 14 03:17:37.892115 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.892090 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202a2300-292b-4bb2-87e0-ce7dfe7526e4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "202a2300-292b-4bb2-87e0-ce7dfe7526e4" (UID: "202a2300-292b-4bb2-87e0-ce7dfe7526e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:17:37.894180 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.894149 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202a2300-292b-4bb2-87e0-ce7dfe7526e4-kube-api-access-5pv5v" (OuterVolumeSpecName: "kube-api-access-5pv5v") pod "202a2300-292b-4bb2-87e0-ce7dfe7526e4" (UID: "202a2300-292b-4bb2-87e0-ce7dfe7526e4"). InnerVolumeSpecName "kube-api-access-5pv5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:17:37.938804 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.938689 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:37.967045 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.967015 2564 generic.go:356] "Generic (PLEG): container finished" podID="202a2300-292b-4bb2-87e0-ce7dfe7526e4" containerID="1581f5e4af418e878ac05e349248678cda73cc5edd922e5a0028f82233995709" exitCode=0 Sep 14 03:17:37.967242 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.967068 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n" Sep 14 03:17:37.967242 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.967100 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n" event={"ID":"202a2300-292b-4bb2-87e0-ce7dfe7526e4","Type":"ContainerDied","Data":"1581f5e4af418e878ac05e349248678cda73cc5edd922e5a0028f82233995709"} Sep 14 03:17:37.967242 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.967163 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n" event={"ID":"202a2300-292b-4bb2-87e0-ce7dfe7526e4","Type":"ContainerDied","Data":"09f9dc93f5d3290c3896b4b4a7d6c3433b04b59ca71b18cf213d5d497b0212ec"} Sep 14 03:17:37.967242 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.967194 2564 scope.go:117] "RemoveContainer" containerID="1581f5e4af418e878ac05e349248678cda73cc5edd922e5a0028f82233995709" Sep 14 03:17:37.980077 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.980058 2564 scope.go:117] "RemoveContainer" containerID="1581f5e4af418e878ac05e349248678cda73cc5edd922e5a0028f82233995709" Sep 14 03:17:37.980370 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:17:37.980349 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1581f5e4af418e878ac05e349248678cda73cc5edd922e5a0028f82233995709\": container with ID starting with 1581f5e4af418e878ac05e349248678cda73cc5edd922e5a0028f82233995709 not found: ID does not exist" containerID="1581f5e4af418e878ac05e349248678cda73cc5edd922e5a0028f82233995709" Sep 14 03:17:37.980433 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.980379 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1581f5e4af418e878ac05e349248678cda73cc5edd922e5a0028f82233995709"} err="failed to get container status \"1581f5e4af418e878ac05e349248678cda73cc5edd922e5a0028f82233995709\": rpc error: code = NotFound desc = could not find container \"1581f5e4af418e878ac05e349248678cda73cc5edd922e5a0028f82233995709\": container with ID starting with 1581f5e4af418e878ac05e349248678cda73cc5edd922e5a0028f82233995709 not found: ID does not exist" Sep 14 03:17:37.993038 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.993013 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/202a2300-292b-4bb2-87e0-ce7dfe7526e4-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:17:37.993038 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.993035 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pv5v\" (UniqueName: \"kubernetes.io/projected/202a2300-292b-4bb2-87e0-ce7dfe7526e4-kube-api-access-5pv5v\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:17:37.993233 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.993214 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n"] Sep 14 03:17:37.999846 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:37.999825 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-9f77a9-7cb54c9b7d-tlh6n"] Sep 14 03:17:38.546205 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:38.546175 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:38.546465 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:38.546447 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:38.547217 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:38.547202 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:38.750936 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:38.750906 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="202a2300-292b-4bb2-87e0-ce7dfe7526e4" path="/var/lib/kubelet/pods/202a2300-292b-4bb2-87e0-ce7dfe7526e4/volumes" Sep 14 03:17:38.751296 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:38.751170 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a8a1eb9-90ad-41f6-9236-f51d2fc75e54" path="/var/lib/kubelet/pods/4a8a1eb9-90ad-41f6-9236-f51d2fc75e54/volumes" Sep 14 03:17:38.751418 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:38.751405 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe754cc5-a71e-4088-a9d6-618fbb31d97f" path="/var/lib/kubelet/pods/fe754cc5-a71e-4088-a9d6-618fbb31d97f/volumes" Sep 14 03:17:38.768387 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:38.768356 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:39.559469 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:39.559435 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:40.261678 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:40.261641 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:41.032541 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:41.032504 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:41.689027 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:41.688995 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:42.402835 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:42.402798 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:42.440203 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:42.440171 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2"] Sep 14 03:17:42.440691 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:42.440657 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2" podUID="6ab284e8-e85c-49d2-b959-1fc69c076cb9" containerName="payload-pre-processing" containerID="cri-o://ed1407efe744cf78de1224413501cab2410eb9c52235b235bf286a04ecda123e" gracePeriod=30 Sep 14 03:17:42.512733 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:42.512692 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:42.626996 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:42.626861 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:42.699111 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:42.699079 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35f4b-tzlzj"] Sep 14 03:17:42.704064 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:42.704038 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-a35f4b-tzlzj"] Sep 14 03:17:42.705109 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:42.705089 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2" Sep 14 03:17:42.731547 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:42.731418 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:42.735826 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:42.735791 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncx6f\" (UniqueName: \"kubernetes.io/projected/6ab284e8-e85c-49d2-b959-1fc69c076cb9-kube-api-access-ncx6f\") pod \"6ab284e8-e85c-49d2-b959-1fc69c076cb9\" (UID: \"6ab284e8-e85c-49d2-b959-1fc69c076cb9\") " Sep 14 03:17:42.735976 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:42.735879 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab284e8-e85c-49d2-b959-1fc69c076cb9-plugins-config-volume\") pod \"6ab284e8-e85c-49d2-b959-1fc69c076cb9\" (UID: \"6ab284e8-e85c-49d2-b959-1fc69c076cb9\") " Sep 14 03:17:42.736273 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:42.736247 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab284e8-e85c-49d2-b959-1fc69c076cb9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6ab284e8-e85c-49d2-b959-1fc69c076cb9" (UID: "6ab284e8-e85c-49d2-b959-1fc69c076cb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:17:42.738179 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:42.738147 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab284e8-e85c-49d2-b959-1fc69c076cb9-kube-api-access-ncx6f" (OuterVolumeSpecName: "kube-api-access-ncx6f") pod "6ab284e8-e85c-49d2-b959-1fc69c076cb9" (UID: "6ab284e8-e85c-49d2-b959-1fc69c076cb9"). InnerVolumeSpecName "kube-api-access-ncx6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:17:42.752666 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:42.752633 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6fc5e8d-ec6c-47b7-ae8e-22f7f7b94216" path="/var/lib/kubelet/pods/c6fc5e8d-ec6c-47b7-ae8e-22f7f7b94216/volumes" Sep 14 03:17:42.837326 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:42.837289 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncx6f\" (UniqueName: \"kubernetes.io/projected/6ab284e8-e85c-49d2-b959-1fc69c076cb9-kube-api-access-ncx6f\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:17:42.837326 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:42.837323 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab284e8-e85c-49d2-b959-1fc69c076cb9-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:17:42.998364 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:42.998269 2564 generic.go:356] "Generic (PLEG): container finished" podID="6ab284e8-e85c-49d2-b959-1fc69c076cb9" containerID="ed1407efe744cf78de1224413501cab2410eb9c52235b235bf286a04ecda123e" exitCode=0 Sep 14 03:17:42.998364 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:42.998325 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2" Sep 14 03:17:42.998641 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:42.998363 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2" event={"ID":"6ab284e8-e85c-49d2-b959-1fc69c076cb9","Type":"ContainerDied","Data":"ed1407efe744cf78de1224413501cab2410eb9c52235b235bf286a04ecda123e"} Sep 14 03:17:42.998641 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:42.998425 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2" event={"ID":"6ab284e8-e85c-49d2-b959-1fc69c076cb9","Type":"ContainerDied","Data":"6c5acf73bdd44d9b37f73f79e8682e14496609a892f9d69b44bf4e2707014104"} Sep 14 03:17:42.998641 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:42.998456 2564 scope.go:117] "RemoveContainer" containerID="ed1407efe744cf78de1224413501cab2410eb9c52235b235bf286a04ecda123e" Sep 14 03:17:43.007945 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:43.007925 2564 scope.go:117] "RemoveContainer" containerID="ed1407efe744cf78de1224413501cab2410eb9c52235b235bf286a04ecda123e" Sep 14 03:17:43.008253 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:17:43.008232 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed1407efe744cf78de1224413501cab2410eb9c52235b235bf286a04ecda123e\": container with ID starting with ed1407efe744cf78de1224413501cab2410eb9c52235b235bf286a04ecda123e not found: ID does not exist" containerID="ed1407efe744cf78de1224413501cab2410eb9c52235b235bf286a04ecda123e" Sep 14 03:17:43.008302 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:43.008261 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed1407efe744cf78de1224413501cab2410eb9c52235b235bf286a04ecda123e"} err="failed to get container status \"ed1407efe744cf78de1224413501cab2410eb9c52235b235bf286a04ecda123e\": rpc error: code = NotFound desc = could not find container \"ed1407efe744cf78de1224413501cab2410eb9c52235b235bf286a04ecda123e\": container with ID starting with ed1407efe744cf78de1224413501cab2410eb9c52235b235bf286a04ecda123e not found: ID does not exist" Sep 14 03:17:43.021747 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:43.021720 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2"] Sep 14 03:17:43.034455 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:43.034427 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a35f4b-555d895794-7knt2"] Sep 14 03:17:43.076409 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:43.076379 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:43.159578 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:43.159544 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:43.834687 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:43.834652 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:44.537848 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:44.537798 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:44.750442 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:44.750408 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ab284e8-e85c-49d2-b959-1fc69c076cb9" path="/var/lib/kubelet/pods/6ab284e8-e85c-49d2-b959-1fc69c076cb9/volumes" Sep 14 03:17:44.786615 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:44.786581 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:44.887599 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:44.887511 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:45.045420 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:45.045374 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:45.487002 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:45.486965 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:46.232350 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:46.232313 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:49.518615 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:49.518574 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:49.621944 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:49.621900 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:49.769617 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:49.769546 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf"] Sep 14 03:17:49.769765 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:49.769743 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" podUID="cc6ff6da-345b-4e30-b748-723c74edaca6" containerName="istio-proxy" containerID="cri-o://99e68ccb63708c048b0c8bdba3f635d5c067636d5fa1ddc81d3832faf18dd2b0" gracePeriod=30 Sep 14 03:17:49.805123 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:49.805087 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:53.549067 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:53.549028 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 03:17:53.549436 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:53.549098 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" podUID="cc6ff6da-345b-4e30-b748-723c74edaca6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 03:17:53.629538 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:53.629494 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:54.526534 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:54.526501 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:55.026822 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.026799 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:55.066409 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.066348 2564 generic.go:356] "Generic (PLEG): container finished" podID="cc6ff6da-345b-4e30-b748-723c74edaca6" containerID="99e68ccb63708c048b0c8bdba3f635d5c067636d5fa1ddc81d3832faf18dd2b0" exitCode=0 Sep 14 03:17:55.066554 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.066405 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" Sep 14 03:17:55.066554 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.066431 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" event={"ID":"cc6ff6da-345b-4e30-b748-723c74edaca6","Type":"ContainerDied","Data":"99e68ccb63708c048b0c8bdba3f635d5c067636d5fa1ddc81d3832faf18dd2b0"} Sep 14 03:17:55.066554 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.066500 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf" event={"ID":"cc6ff6da-345b-4e30-b748-723c74edaca6","Type":"ContainerDied","Data":"2abc689f6af3eb0c0bbe39d4405e9a20411e1e4bb3440d3bec94b014335e45da"} Sep 14 03:17:55.066554 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.066525 2564 scope.go:117] "RemoveContainer" containerID="99e68ccb63708c048b0c8bdba3f635d5c067636d5fa1ddc81d3832faf18dd2b0" Sep 14 03:17:55.075408 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.075394 2564 scope.go:117] "RemoveContainer" containerID="907314fbb654eac142998d522d20c933820a3118ba4b04b41605b23ea6176fc8" Sep 14 03:17:55.085309 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.085289 2564 scope.go:117] "RemoveContainer" containerID="99e68ccb63708c048b0c8bdba3f635d5c067636d5fa1ddc81d3832faf18dd2b0" Sep 14 03:17:55.085610 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:17:55.085594 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99e68ccb63708c048b0c8bdba3f635d5c067636d5fa1ddc81d3832faf18dd2b0\": container with ID starting with 99e68ccb63708c048b0c8bdba3f635d5c067636d5fa1ddc81d3832faf18dd2b0 not found: ID does not exist" containerID="99e68ccb63708c048b0c8bdba3f635d5c067636d5fa1ddc81d3832faf18dd2b0" Sep 14 03:17:55.085681 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.085619 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99e68ccb63708c048b0c8bdba3f635d5c067636d5fa1ddc81d3832faf18dd2b0"} err="failed to get container status \"99e68ccb63708c048b0c8bdba3f635d5c067636d5fa1ddc81d3832faf18dd2b0\": rpc error: code = NotFound desc = could not find container \"99e68ccb63708c048b0c8bdba3f635d5c067636d5fa1ddc81d3832faf18dd2b0\": container with ID starting with 99e68ccb63708c048b0c8bdba3f635d5c067636d5fa1ddc81d3832faf18dd2b0 not found: ID does not exist" Sep 14 03:17:55.085681 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.085634 2564 scope.go:117] "RemoveContainer" containerID="907314fbb654eac142998d522d20c933820a3118ba4b04b41605b23ea6176fc8" Sep 14 03:17:55.085917 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:17:55.085894 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"907314fbb654eac142998d522d20c933820a3118ba4b04b41605b23ea6176fc8\": container with ID starting with 907314fbb654eac142998d522d20c933820a3118ba4b04b41605b23ea6176fc8 not found: ID does not exist" containerID="907314fbb654eac142998d522d20c933820a3118ba4b04b41605b23ea6176fc8" Sep 14 03:17:55.085962 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.085927 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"907314fbb654eac142998d522d20c933820a3118ba4b04b41605b23ea6176fc8"} err="failed to get container status \"907314fbb654eac142998d522d20c933820a3118ba4b04b41605b23ea6176fc8\": rpc error: code = NotFound desc = could not find container \"907314fbb654eac142998d522d20c933820a3118ba4b04b41605b23ea6176fc8\": container with ID starting with 907314fbb654eac142998d522d20c933820a3118ba4b04b41605b23ea6176fc8 not found: ID does not exist" Sep 14 03:17:55.150567 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.150548 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-workload-certs\") pod \"cc6ff6da-345b-4e30-b748-723c74edaca6\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " Sep 14 03:17:55.150677 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.150578 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-workload-socket\") pod \"cc6ff6da-345b-4e30-b748-723c74edaca6\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " Sep 14 03:17:55.150677 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.150605 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6ff6da-345b-4e30-b748-723c74edaca6-istiod-ca-cert\") pod \"cc6ff6da-345b-4e30-b748-723c74edaca6\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " Sep 14 03:17:55.150677 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.150620 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-credential-socket\") pod \"cc6ff6da-345b-4e30-b748-723c74edaca6\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " Sep 14 03:17:55.150677 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.150645 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-podinfo\") pod \"cc6ff6da-345b-4e30-b748-723c74edaca6\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " Sep 14 03:17:55.150677 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.150669 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-envoy\") pod \"cc6ff6da-345b-4e30-b748-723c74edaca6\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " Sep 14 03:17:55.150902 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.150695 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-token\") pod \"cc6ff6da-345b-4e30-b748-723c74edaca6\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " Sep 14 03:17:55.150902 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.150810 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thzk4\" (UniqueName: \"kubernetes.io/projected/cc6ff6da-345b-4e30-b748-723c74edaca6-kube-api-access-thzk4\") pod \"cc6ff6da-345b-4e30-b748-723c74edaca6\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " Sep 14 03:17:55.150902 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.150825 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cc6ff6da-345b-4e30-b748-723c74edaca6" (UID: "cc6ff6da-345b-4e30-b748-723c74edaca6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:17:55.150902 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.150849 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-data\") pod \"cc6ff6da-345b-4e30-b748-723c74edaca6\" (UID: \"cc6ff6da-345b-4e30-b748-723c74edaca6\") " Sep 14 03:17:55.150902 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.150874 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cc6ff6da-345b-4e30-b748-723c74edaca6" (UID: "cc6ff6da-345b-4e30-b748-723c74edaca6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:17:55.151154 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.150995 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6ff6da-345b-4e30-b748-723c74edaca6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cc6ff6da-345b-4e30-b748-723c74edaca6" (UID: "cc6ff6da-345b-4e30-b748-723c74edaca6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:17:55.151154 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.151137 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-data" (OuterVolumeSpecName: "istio-data") pod "cc6ff6da-345b-4e30-b748-723c74edaca6" (UID: "cc6ff6da-345b-4e30-b748-723c74edaca6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:17:55.151290 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.151270 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-data\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:17:55.151364 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.151292 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-workload-certs\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:17:55.151364 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.151303 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-workload-socket\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:17:55.151364 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.151311 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6ff6da-345b-4e30-b748-723c74edaca6-istiod-ca-cert\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:17:55.151364 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.151265 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cc6ff6da-345b-4e30-b748-723c74edaca6" (UID: "cc6ff6da-345b-4e30-b748-723c74edaca6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:17:55.152883 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.152856 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6ff6da-345b-4e30-b748-723c74edaca6-kube-api-access-thzk4" (OuterVolumeSpecName: "kube-api-access-thzk4") pod "cc6ff6da-345b-4e30-b748-723c74edaca6" (UID: "cc6ff6da-345b-4e30-b748-723c74edaca6"). InnerVolumeSpecName "kube-api-access-thzk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:17:55.153126 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.153106 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cc6ff6da-345b-4e30-b748-723c74edaca6" (UID: "cc6ff6da-345b-4e30-b748-723c74edaca6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 03:17:55.153398 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.153379 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cc6ff6da-345b-4e30-b748-723c74edaca6" (UID: "cc6ff6da-345b-4e30-b748-723c74edaca6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:17:55.153508 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.153405 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-token" (OuterVolumeSpecName: "istio-token") pod "cc6ff6da-345b-4e30-b748-723c74edaca6" (UID: "cc6ff6da-345b-4e30-b748-723c74edaca6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:17:55.252327 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.252299 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-credential-socket\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:17:55.252327 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.252323 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-podinfo\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:17:55.252471 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.252336 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-envoy\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:17:55.252471 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.252348 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc6ff6da-345b-4e30-b748-723c74edaca6-istio-token\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:17:55.252471 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.252359 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thzk4\" (UniqueName: \"kubernetes.io/projected/cc6ff6da-345b-4e30-b748-723c74edaca6-kube-api-access-thzk4\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:17:55.255426 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.255398 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:55.396558 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.396524 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf"] Sep 14 03:17:55.401360 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:55.401337 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-a35f4b-openshift-default-7d75dbcbf8-s64nf"] Sep 14 03:17:56.087711 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:56.087675 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:56.750702 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:56.750670 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc6ff6da-345b-4e30-b748-723c74edaca6" path="/var/lib/kubelet/pods/cc6ff6da-345b-4e30-b748-723c74edaca6/volumes" Sep 14 03:17:56.843759 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:56.843716 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:17:57.588523 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:17:57.588464 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:18:08.940763 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:08.940726 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:18:08.963023 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:08.962993 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9"] Sep 14 03:18:08.963304 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:08.963274 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9" podUID="3fada7bb-85e2-4680-ab7b-e7dde698eca3" containerName="payload-processing" containerID="cri-o://3a93beaadf6409b642bbd8dc54d5cf9605393cf94a8da27bcc96ae2982600c52" gracePeriod=30 Sep 14 03:18:08.972544 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:08.972509 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9"] Sep 14 03:18:08.972740 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:08.972717 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9" podUID="02aaca25-da12-4173-adc6-a71fdfd95346" containerName="payload-pre-processing" containerID="cri-o://30037c5af164f8f27fd9a5f6c860e5b02f0017edc3b115b7f0709d723e676c60" gracePeriod=30 Sep 14 03:18:09.016105 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.016077 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:18:09.099216 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.099162 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:18:09.151055 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.151017 2564 generic.go:356] "Generic (PLEG): container finished" podID="02aaca25-da12-4173-adc6-a71fdfd95346" containerID="30037c5af164f8f27fd9a5f6c860e5b02f0017edc3b115b7f0709d723e676c60" exitCode=0 Sep 14 03:18:09.151209 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.151080 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9" event={"ID":"02aaca25-da12-4173-adc6-a71fdfd95346","Type":"ContainerDied","Data":"30037c5af164f8f27fd9a5f6c860e5b02f0017edc3b115b7f0709d723e676c60"} Sep 14 03:18:09.152904 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.152883 2564 generic.go:356] "Generic (PLEG): container finished" podID="3fada7bb-85e2-4680-ab7b-e7dde698eca3" containerID="3a93beaadf6409b642bbd8dc54d5cf9605393cf94a8da27bcc96ae2982600c52" exitCode=0 Sep 14 03:18:09.153032 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.152929 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9" event={"ID":"3fada7bb-85e2-4680-ab7b-e7dde698eca3","Type":"ContainerDied","Data":"3a93beaadf6409b642bbd8dc54d5cf9605393cf94a8da27bcc96ae2982600c52"} Sep 14 03:18:09.191951 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.191799 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:18:09.262921 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.262890 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:18:09.270046 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.270024 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9" Sep 14 03:18:09.273233 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.273214 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9" Sep 14 03:18:09.373498 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.373446 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02aaca25-da12-4173-adc6-a71fdfd95346-plugins-config-volume\") pod \"02aaca25-da12-4173-adc6-a71fdfd95346\" (UID: \"02aaca25-da12-4173-adc6-a71fdfd95346\") " Sep 14 03:18:09.373498 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.373506 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chsbq\" (UniqueName: \"kubernetes.io/projected/02aaca25-da12-4173-adc6-a71fdfd95346-kube-api-access-chsbq\") pod \"02aaca25-da12-4173-adc6-a71fdfd95346\" (UID: \"02aaca25-da12-4173-adc6-a71fdfd95346\") " Sep 14 03:18:09.373742 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.373525 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsv7g\" (UniqueName: \"kubernetes.io/projected/3fada7bb-85e2-4680-ab7b-e7dde698eca3-kube-api-access-fsv7g\") pod \"3fada7bb-85e2-4680-ab7b-e7dde698eca3\" (UID: \"3fada7bb-85e2-4680-ab7b-e7dde698eca3\") " Sep 14 03:18:09.373742 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.373570 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fada7bb-85e2-4680-ab7b-e7dde698eca3-plugins-config-volume\") pod \"3fada7bb-85e2-4680-ab7b-e7dde698eca3\" (UID: \"3fada7bb-85e2-4680-ab7b-e7dde698eca3\") " Sep 14 03:18:09.373988 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.373842 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02aaca25-da12-4173-adc6-a71fdfd95346-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "02aaca25-da12-4173-adc6-a71fdfd95346" (UID: "02aaca25-da12-4173-adc6-a71fdfd95346"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:18:09.374099 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.373985 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fada7bb-85e2-4680-ab7b-e7dde698eca3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3fada7bb-85e2-4680-ab7b-e7dde698eca3" (UID: "3fada7bb-85e2-4680-ab7b-e7dde698eca3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:18:09.375898 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.375875 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fada7bb-85e2-4680-ab7b-e7dde698eca3-kube-api-access-fsv7g" (OuterVolumeSpecName: "kube-api-access-fsv7g") pod "3fada7bb-85e2-4680-ab7b-e7dde698eca3" (UID: "3fada7bb-85e2-4680-ab7b-e7dde698eca3"). InnerVolumeSpecName "kube-api-access-fsv7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:18:09.375974 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.375907 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02aaca25-da12-4173-adc6-a71fdfd95346-kube-api-access-chsbq" (OuterVolumeSpecName: "kube-api-access-chsbq") pod "02aaca25-da12-4173-adc6-a71fdfd95346" (UID: "02aaca25-da12-4173-adc6-a71fdfd95346"). InnerVolumeSpecName "kube-api-access-chsbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:18:09.474611 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.474534 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02aaca25-da12-4173-adc6-a71fdfd95346-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:18:09.474611 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.474557 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chsbq\" (UniqueName: \"kubernetes.io/projected/02aaca25-da12-4173-adc6-a71fdfd95346-kube-api-access-chsbq\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:18:09.474611 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.474569 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsv7g\" (UniqueName: \"kubernetes.io/projected/3fada7bb-85e2-4680-ab7b-e7dde698eca3-kube-api-access-fsv7g\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:18:09.474611 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.474577 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fada7bb-85e2-4680-ab7b-e7dde698eca3-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:18:09.564225 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.564197 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:18:09.665656 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:09.665625 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:18:10.159561 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:10.159529 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9" Sep 14 03:18:10.159990 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:10.159584 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9" event={"ID":"02aaca25-da12-4173-adc6-a71fdfd95346","Type":"ContainerDied","Data":"2429c0853e30edc4b7b1ce6ebe17c86e697502fb58a72cab9962ce9e45caa766"} Sep 14 03:18:10.159990 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:10.159640 2564 scope.go:117] "RemoveContainer" containerID="30037c5af164f8f27fd9a5f6c860e5b02f0017edc3b115b7f0709d723e676c60" Sep 14 03:18:10.161101 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:10.161082 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9" event={"ID":"3fada7bb-85e2-4680-ab7b-e7dde698eca3","Type":"ContainerDied","Data":"7a258e272caa4eb8a2af50de03aa05a8c453fc96e9ef8f0b3ab75e323f9a2be9"} Sep 14 03:18:10.161101 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:10.161099 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9" Sep 14 03:18:10.169414 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:10.169396 2564 scope.go:117] "RemoveContainer" containerID="3a93beaadf6409b642bbd8dc54d5cf9605393cf94a8da27bcc96ae2982600c52" Sep 14 03:18:10.190257 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:10.190234 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9"] Sep 14 03:18:10.197829 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:10.197810 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2611af-7cdddf7dc8-dffp9"] Sep 14 03:18:10.212152 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:10.212126 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9"] Sep 14 03:18:10.218698 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:10.218678 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2611af-76c9b5d55f-rh4n9"] Sep 14 03:18:10.413531 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:10.413460 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:18:10.750683 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:10.750609 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02aaca25-da12-4173-adc6-a71fdfd95346" path="/var/lib/kubelet/pods/02aaca25-da12-4173-adc6-a71fdfd95346/volumes" Sep 14 03:18:10.750978 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:10.750964 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fada7bb-85e2-4680-ab7b-e7dde698eca3" path="/var/lib/kubelet/pods/3fada7bb-85e2-4680-ab7b-e7dde698eca3/volumes" Sep 14 03:18:11.222377 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:11.222339 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:18:11.805257 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:11.805227 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:18:14.638416 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:14.638386 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:18:49.012277 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:49.012239 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:18:51.558688 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:51.558607 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:18:51.935986 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:51.935958 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:18:58.196750 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:18:58.196699 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:01.702058 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:01.702014 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:03.898161 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:03.898118 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:03.998700 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:03.998669 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:04.153809 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.153718 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:04.249942 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.249902 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:04.438120 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.438031 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:04.763871 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.763783 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdf5b3-6b6m9"] Sep 14 03:19:04.764255 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.764236 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02aaca25-da12-4173-adc6-a71fdfd95346" containerName="payload-pre-processing" Sep 14 03:19:04.764347 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.764259 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="02aaca25-da12-4173-adc6-a71fdfd95346" containerName="payload-pre-processing" Sep 14 03:19:04.764347 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.764272 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fada7bb-85e2-4680-ab7b-e7dde698eca3" containerName="payload-processing" Sep 14 03:19:04.764347 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.764281 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fada7bb-85e2-4680-ab7b-e7dde698eca3" containerName="payload-processing" Sep 14 03:19:04.764347 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.764297 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc6ff6da-345b-4e30-b748-723c74edaca6" containerName="istio-proxy" Sep 14 03:19:04.764347 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.764305 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc6ff6da-345b-4e30-b748-723c74edaca6" containerName="istio-proxy" Sep 14 03:19:04.764347 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.764320 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6fc5e8d-ec6c-47b7-ae8e-22f7f7b94216" containerName="revoke-keys" Sep 14 03:19:04.764347 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.764328 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6fc5e8d-ec6c-47b7-ae8e-22f7f7b94216" containerName="revoke-keys" Sep 14 03:19:04.764741 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.764359 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="202a2300-292b-4bb2-87e0-ce7dfe7526e4" containerName="payload-pre-processing" Sep 14 03:19:04.764741 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.764368 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="202a2300-292b-4bb2-87e0-ce7dfe7526e4" containerName="payload-pre-processing" Sep 14 03:19:04.764741 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.764377 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ab284e8-e85c-49d2-b959-1fc69c076cb9" containerName="payload-pre-processing" Sep 14 03:19:04.764741 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.764385 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ab284e8-e85c-49d2-b959-1fc69c076cb9" containerName="payload-pre-processing" Sep 14 03:19:04.764741 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.764530 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc6ff6da-345b-4e30-b748-723c74edaca6" containerName="istio-proxy" Sep 14 03:19:04.764741 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.764544 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc6ff6da-345b-4e30-b748-723c74edaca6" containerName="istio-proxy" Sep 14 03:19:04.764741 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.764555 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ab284e8-e85c-49d2-b959-1fc69c076cb9" containerName="payload-pre-processing" Sep 14 03:19:04.764741 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.764566 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6fc5e8d-ec6c-47b7-ae8e-22f7f7b94216" containerName="revoke-keys" Sep 14 03:19:04.764741 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.764576 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fada7bb-85e2-4680-ab7b-e7dde698eca3" containerName="payload-processing" Sep 14 03:19:04.764741 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.764587 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="202a2300-292b-4bb2-87e0-ce7dfe7526e4" containerName="payload-pre-processing" Sep 14 03:19:04.764741 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.764596 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="02aaca25-da12-4173-adc6-a71fdfd95346" containerName="payload-pre-processing" Sep 14 03:19:04.767775 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.767754 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdf5b3-6b6m9" Sep 14 03:19:04.770204 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.770182 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 03:19:04.770324 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.770239 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4hf5p\"" Sep 14 03:19:04.775418 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.775089 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdf5b3-6b6m9"] Sep 14 03:19:04.856939 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.856901 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef0e3bc4-6eff-4176-9825-168c33aa9c03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bdf5b3-6b6m9\" (UID: \"ef0e3bc4-6eff-4176-9825-168c33aa9c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdf5b3-6b6m9" Sep 14 03:19:04.958195 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.958155 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef0e3bc4-6eff-4176-9825-168c33aa9c03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bdf5b3-6b6m9\" (UID: \"ef0e3bc4-6eff-4176-9825-168c33aa9c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdf5b3-6b6m9" Sep 14 03:19:04.958836 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:04.958814 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef0e3bc4-6eff-4176-9825-168c33aa9c03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bdf5b3-6b6m9\" (UID: \"ef0e3bc4-6eff-4176-9825-168c33aa9c03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdf5b3-6b6m9" Sep 14 03:19:05.080173 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:05.080072 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdf5b3-6b6m9" Sep 14 03:19:05.145560 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:05.145429 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:05.356874 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:05.353750 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdf5b3-6b6m9"] Sep 14 03:19:05.360294 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:19:05.360246 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef0e3bc4_6eff_4176_9825_168c33aa9c03.slice/crio-8c3e8f565d857d97fc7575da864dc6a83e4f9a6c23de759dd4bfc50421692b18 WatchSource:0}: Error finding container 8c3e8f565d857d97fc7575da864dc6a83e4f9a6c23de759dd4bfc50421692b18: Status 404 returned error can't find the container with id 8c3e8f565d857d97fc7575da864dc6a83e4f9a6c23de759dd4bfc50421692b18 Sep 14 03:19:05.457627 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:05.457589 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdf5b3-6b6m9" event={"ID":"ef0e3bc4-6eff-4176-9825-168c33aa9c03","Type":"ContainerStarted","Data":"8c3e8f565d857d97fc7575da864dc6a83e4f9a6c23de759dd4bfc50421692b18"} Sep 14 03:19:05.822271 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:05.822231 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:06.451812 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:06.451706 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:06.464843 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:06.464804 2564 generic.go:356] "Generic (PLEG): container finished" podID="ef0e3bc4-6eff-4176-9825-168c33aa9c03" containerID="e3ea54ff51fb5ca13c0489bed6dad7c3559d107fa2479656814f78ee7ab5fc71" exitCode=0 Sep 14 03:19:06.465029 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:06.464849 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdf5b3-6b6m9" event={"ID":"ef0e3bc4-6eff-4176-9825-168c33aa9c03","Type":"ContainerDied","Data":"e3ea54ff51fb5ca13c0489bed6dad7c3559d107fa2479656814f78ee7ab5fc71"} Sep 14 03:19:07.107117 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:07.107071 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:07.601647 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:07.601619 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdf5b3-6b6m9" Sep 14 03:19:07.686060 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:07.686021 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef0e3bc4-6eff-4176-9825-168c33aa9c03-maas-api-service-ca\") pod \"ef0e3bc4-6eff-4176-9825-168c33aa9c03\" (UID: \"ef0e3bc4-6eff-4176-9825-168c33aa9c03\") " Sep 14 03:19:07.686388 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:07.686364 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef0e3bc4-6eff-4176-9825-168c33aa9c03-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ef0e3bc4-6eff-4176-9825-168c33aa9c03" (UID: "ef0e3bc4-6eff-4176-9825-168c33aa9c03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:19:07.771157 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:07.771076 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:07.786814 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:07.786781 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef0e3bc4-6eff-4176-9825-168c33aa9c03-maas-api-service-ca\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:19:08.476716 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:08.476681 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdf5b3-6b6m9" event={"ID":"ef0e3bc4-6eff-4176-9825-168c33aa9c03","Type":"ContainerDied","Data":"8c3e8f565d857d97fc7575da864dc6a83e4f9a6c23de759dd4bfc50421692b18"} Sep 14 03:19:08.476926 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:08.476731 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c3e8f565d857d97fc7575da864dc6a83e4f9a6c23de759dd4bfc50421692b18" Sep 14 03:19:08.476926 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:08.476694 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdf5b3-6b6m9" Sep 14 03:19:15.290094 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:15.290055 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:15.359266 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:15.359232 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:15.426212 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:15.426166 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:15.482110 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:15.482063 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdf5b3-6b6m9"] Sep 14 03:19:15.486338 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:15.486308 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdf5b3-6b6m9"] Sep 14 03:19:15.491333 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:15.491297 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:15.861150 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:15.861114 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:15.961520 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:15.961488 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:16.418370 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:16.418338 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:16.750621 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:16.750543 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef0e3bc4-6eff-4176-9825-168c33aa9c03" path="/var/lib/kubelet/pods/ef0e3bc4-6eff-4176-9825-168c33aa9c03/volumes" Sep 14 03:19:17.194069 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:17.194033 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:22.267036 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:22.267003 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:22.367602 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:22.367555 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:22.575750 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:22.575659 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:23.172358 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:23.172324 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:23.911750 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:23.911713 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:24.577795 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:24.577755 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:25.360624 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:25.360587 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:33.529235 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:33.529205 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:33.533928 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:33.533895 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0c919e-29822595-fpqkj"] Sep 14 03:19:33.542925 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:33.542896 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-0c919e-29822595-fpqkj"] Sep 14 03:19:33.572029 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:33.571990 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh"] Sep 14 03:19:33.572248 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:33.572222 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh" podUID="95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc" containerName="payload-processing" containerID="cri-o://94ead2c0f92abb9c333874529fee8b02daec977cfdd8a5b9f98ff44f841e4b96" gracePeriod=30 Sep 14 03:19:33.583219 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:33.583179 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb"] Sep 14 03:19:33.583767 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:33.583735 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb" podUID="3a39416c-586b-4890-9164-cea97b576dfe" containerName="payload-pre-processing" containerID="cri-o://d7412bc20f9cd591596c852c2953259e7272a7e0e947120c037166533520a9bb" gracePeriod=30 Sep 14 03:19:33.631424 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:33.631386 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:33.726929 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:33.726887 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:33.806298 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:33.806267 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:33.841056 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:33.841026 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh" Sep 14 03:19:33.870827 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:33.870799 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb" Sep 14 03:19:33.927634 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:33.927603 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc-plugins-config-volume\") pod \"95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc\" (UID: \"95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc\") " Sep 14 03:19:33.927844 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:33.927656 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59rhd\" (UniqueName: \"kubernetes.io/projected/95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc-kube-api-access-59rhd\") pod \"95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc\" (UID: \"95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc\") " Sep 14 03:19:33.927844 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:33.927686 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rjfh\" (UniqueName: \"kubernetes.io/projected/3a39416c-586b-4890-9164-cea97b576dfe-kube-api-access-2rjfh\") pod \"3a39416c-586b-4890-9164-cea97b576dfe\" (UID: \"3a39416c-586b-4890-9164-cea97b576dfe\") " Sep 14 03:19:33.927844 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:33.927722 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a39416c-586b-4890-9164-cea97b576dfe-plugins-config-volume\") pod \"3a39416c-586b-4890-9164-cea97b576dfe\" (UID: \"3a39416c-586b-4890-9164-cea97b576dfe\") " Sep 14 03:19:33.928162 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:33.928128 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc" (UID: "95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:19:33.928162 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:33.928152 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a39416c-586b-4890-9164-cea97b576dfe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3a39416c-586b-4890-9164-cea97b576dfe" (UID: "3a39416c-586b-4890-9164-cea97b576dfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:19:33.930108 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:33.930081 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc-kube-api-access-59rhd" (OuterVolumeSpecName: "kube-api-access-59rhd") pod "95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc" (UID: "95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc"). InnerVolumeSpecName "kube-api-access-59rhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:19:33.930288 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:33.930268 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a39416c-586b-4890-9164-cea97b576dfe-kube-api-access-2rjfh" (OuterVolumeSpecName: "kube-api-access-2rjfh") pod "3a39416c-586b-4890-9164-cea97b576dfe" (UID: "3a39416c-586b-4890-9164-cea97b576dfe"). InnerVolumeSpecName "kube-api-access-2rjfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:19:34.028519 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.028461 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:19:34.028519 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.028512 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59rhd\" (UniqueName: \"kubernetes.io/projected/95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc-kube-api-access-59rhd\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:19:34.028519 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.028526 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rjfh\" (UniqueName: \"kubernetes.io/projected/3a39416c-586b-4890-9164-cea97b576dfe-kube-api-access-2rjfh\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:19:34.028766 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.028538 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a39416c-586b-4890-9164-cea97b576dfe-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:19:34.148331 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.148296 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:34.247775 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.247737 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:34.622405 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.622370 2564 generic.go:356] "Generic (PLEG): container finished" podID="95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc" containerID="94ead2c0f92abb9c333874529fee8b02daec977cfdd8a5b9f98ff44f841e4b96" exitCode=0 Sep 14 03:19:34.622976 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.622418 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh" Sep 14 03:19:34.622976 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.622446 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh" event={"ID":"95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc","Type":"ContainerDied","Data":"94ead2c0f92abb9c333874529fee8b02daec977cfdd8a5b9f98ff44f841e4b96"} Sep 14 03:19:34.622976 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.622578 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh" event={"ID":"95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc","Type":"ContainerDied","Data":"8e85a9a936c6e919d1c75587e297a1850c46050d581e7116456e706011fed60d"} Sep 14 03:19:34.622976 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.622616 2564 scope.go:117] "RemoveContainer" containerID="94ead2c0f92abb9c333874529fee8b02daec977cfdd8a5b9f98ff44f841e4b96" Sep 14 03:19:34.624134 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.624098 2564 generic.go:356] "Generic (PLEG): container finished" podID="3a39416c-586b-4890-9164-cea97b576dfe" containerID="d7412bc20f9cd591596c852c2953259e7272a7e0e947120c037166533520a9bb" exitCode=0 Sep 14 03:19:34.624224 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.624147 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb" Sep 14 03:19:34.624224 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.624184 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb" event={"ID":"3a39416c-586b-4890-9164-cea97b576dfe","Type":"ContainerDied","Data":"d7412bc20f9cd591596c852c2953259e7272a7e0e947120c037166533520a9bb"} Sep 14 03:19:34.624331 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.624302 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb" event={"ID":"3a39416c-586b-4890-9164-cea97b576dfe","Type":"ContainerDied","Data":"82658ee2d50f3e6866343f221e393f74396207cf5f35f55d72870af801a9f2b2"} Sep 14 03:19:34.632597 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.632578 2564 scope.go:117] "RemoveContainer" containerID="94ead2c0f92abb9c333874529fee8b02daec977cfdd8a5b9f98ff44f841e4b96" Sep 14 03:19:34.632923 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:19:34.632906 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94ead2c0f92abb9c333874529fee8b02daec977cfdd8a5b9f98ff44f841e4b96\": container with ID starting with 94ead2c0f92abb9c333874529fee8b02daec977cfdd8a5b9f98ff44f841e4b96 not found: ID does not exist" containerID="94ead2c0f92abb9c333874529fee8b02daec977cfdd8a5b9f98ff44f841e4b96" Sep 14 03:19:34.632978 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.632932 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94ead2c0f92abb9c333874529fee8b02daec977cfdd8a5b9f98ff44f841e4b96"} err="failed to get container status \"94ead2c0f92abb9c333874529fee8b02daec977cfdd8a5b9f98ff44f841e4b96\": rpc error: code = NotFound desc = could not find container \"94ead2c0f92abb9c333874529fee8b02daec977cfdd8a5b9f98ff44f841e4b96\": container with ID starting with 94ead2c0f92abb9c333874529fee8b02daec977cfdd8a5b9f98ff44f841e4b96 not found: ID does not exist" Sep 14 03:19:34.632978 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.632951 2564 scope.go:117] "RemoveContainer" containerID="d7412bc20f9cd591596c852c2953259e7272a7e0e947120c037166533520a9bb" Sep 14 03:19:34.641555 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.641538 2564 scope.go:117] "RemoveContainer" containerID="d7412bc20f9cd591596c852c2953259e7272a7e0e947120c037166533520a9bb" Sep 14 03:19:34.641870 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:19:34.641855 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7412bc20f9cd591596c852c2953259e7272a7e0e947120c037166533520a9bb\": container with ID starting with d7412bc20f9cd591596c852c2953259e7272a7e0e947120c037166533520a9bb not found: ID does not exist" containerID="d7412bc20f9cd591596c852c2953259e7272a7e0e947120c037166533520a9bb" Sep 14 03:19:34.641914 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.641877 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7412bc20f9cd591596c852c2953259e7272a7e0e947120c037166533520a9bb"} err="failed to get container status \"d7412bc20f9cd591596c852c2953259e7272a7e0e947120c037166533520a9bb\": rpc error: code = NotFound desc = could not find container \"d7412bc20f9cd591596c852c2953259e7272a7e0e947120c037166533520a9bb\": container with ID starting with d7412bc20f9cd591596c852c2953259e7272a7e0e947120c037166533520a9bb not found: ID does not exist" Sep 14 03:19:34.647773 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.647749 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh"] Sep 14 03:19:34.652188 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.652161 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-0c919e-647b5bf5ff-l54lh"] Sep 14 03:19:34.662639 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.662614 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb"] Sep 14 03:19:34.669773 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.669749 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-0c919e-54c556dddb-c9qxb"] Sep 14 03:19:34.749991 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.749959 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a39416c-586b-4890-9164-cea97b576dfe" path="/var/lib/kubelet/pods/3a39416c-586b-4890-9164-cea97b576dfe/volumes" Sep 14 03:19:34.750266 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.750251 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6935df75-6d19-4636-b2a4-354b1c1bdd58" path="/var/lib/kubelet/pods/6935df75-6d19-4636-b2a4-354b1c1bdd58/volumes" Sep 14 03:19:34.750562 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.750550 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc" path="/var/lib/kubelet/pods/95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc/volumes" Sep 14 03:19:34.892076 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:34.891984 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:39.259993 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:39.259948 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:40.559748 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:40.559712 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:40.661141 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:40.661103 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:40.816463 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:40.816374 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz"] Sep 14 03:19:40.816677 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:40.816649 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" podUID="2e8a7b82-b28d-49b1-b806-9a53e60391df" containerName="istio-proxy" containerID="cri-o://b9b1fd12b8a7be60f48da215872265594ed948c359d8e767a29ba6e67f4d3413" gracePeriod=30 Sep 14 03:19:40.834439 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:40.834406 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:42.163549 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:42.163519 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:42.759578 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:42.759532 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:46.069419 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.069390 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:19:46.148548 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.148512 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-podinfo\") pod \"2e8a7b82-b28d-49b1-b806-9a53e60391df\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " Sep 14 03:19:46.148723 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.148588 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-data\") pod \"2e8a7b82-b28d-49b1-b806-9a53e60391df\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " Sep 14 03:19:46.148723 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.148611 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-workload-certs\") pod \"2e8a7b82-b28d-49b1-b806-9a53e60391df\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " Sep 14 03:19:46.148723 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.148677 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-envoy\") pod \"2e8a7b82-b28d-49b1-b806-9a53e60391df\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " Sep 14 03:19:46.148723 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.148708 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vk7c9\" (UniqueName: \"kubernetes.io/projected/2e8a7b82-b28d-49b1-b806-9a53e60391df-kube-api-access-vk7c9\") pod \"2e8a7b82-b28d-49b1-b806-9a53e60391df\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " Sep 14 03:19:46.148911 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.148751 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-token\") pod \"2e8a7b82-b28d-49b1-b806-9a53e60391df\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " Sep 14 03:19:46.148911 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.148783 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8a7b82-b28d-49b1-b806-9a53e60391df-istiod-ca-cert\") pod \"2e8a7b82-b28d-49b1-b806-9a53e60391df\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " Sep 14 03:19:46.148911 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.148805 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-workload-socket\") pod \"2e8a7b82-b28d-49b1-b806-9a53e60391df\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " Sep 14 03:19:46.148911 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.148842 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-credential-socket\") pod \"2e8a7b82-b28d-49b1-b806-9a53e60391df\" (UID: \"2e8a7b82-b28d-49b1-b806-9a53e60391df\") " Sep 14 03:19:46.149122 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.148918 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-data" (OuterVolumeSpecName: "istio-data") pod "2e8a7b82-b28d-49b1-b806-9a53e60391df" (UID: "2e8a7b82-b28d-49b1-b806-9a53e60391df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:19:46.149122 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.149001 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2e8a7b82-b28d-49b1-b806-9a53e60391df" (UID: "2e8a7b82-b28d-49b1-b806-9a53e60391df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:19:46.149232 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.149187 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2e8a7b82-b28d-49b1-b806-9a53e60391df" (UID: "2e8a7b82-b28d-49b1-b806-9a53e60391df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:19:46.149320 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.149291 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2e8a7b82-b28d-49b1-b806-9a53e60391df" (UID: "2e8a7b82-b28d-49b1-b806-9a53e60391df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:19:46.149533 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.149517 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-credential-socket\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:19:46.149585 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.149539 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-data\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:19:46.149585 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.149556 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-workload-certs\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:19:46.149585 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.149535 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e8a7b82-b28d-49b1-b806-9a53e60391df-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2e8a7b82-b28d-49b1-b806-9a53e60391df" (UID: "2e8a7b82-b28d-49b1-b806-9a53e60391df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:19:46.151053 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.151017 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2e8a7b82-b28d-49b1-b806-9a53e60391df" (UID: "2e8a7b82-b28d-49b1-b806-9a53e60391df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:19:46.151200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.151103 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2e8a7b82-b28d-49b1-b806-9a53e60391df" (UID: "2e8a7b82-b28d-49b1-b806-9a53e60391df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 03:19:46.151471 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.151447 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e8a7b82-b28d-49b1-b806-9a53e60391df-kube-api-access-vk7c9" (OuterVolumeSpecName: "kube-api-access-vk7c9") pod "2e8a7b82-b28d-49b1-b806-9a53e60391df" (UID: "2e8a7b82-b28d-49b1-b806-9a53e60391df"). InnerVolumeSpecName "kube-api-access-vk7c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:19:46.151471 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.151459 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-token" (OuterVolumeSpecName: "istio-token") pod "2e8a7b82-b28d-49b1-b806-9a53e60391df" (UID: "2e8a7b82-b28d-49b1-b806-9a53e60391df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:19:46.250541 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.250505 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-token\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:19:46.250541 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.250535 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8a7b82-b28d-49b1-b806-9a53e60391df-istiod-ca-cert\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:19:46.250541 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.250544 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-workload-socket\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:19:46.250785 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.250554 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-podinfo\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:19:46.250785 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.250562 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8a7b82-b28d-49b1-b806-9a53e60391df-istio-envoy\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:19:46.250785 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.250570 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk7c9\" (UniqueName: \"kubernetes.io/projected/2e8a7b82-b28d-49b1-b806-9a53e60391df-kube-api-access-vk7c9\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:19:46.695016 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.694982 2564 generic.go:356] "Generic (PLEG): container finished" podID="2e8a7b82-b28d-49b1-b806-9a53e60391df" containerID="b9b1fd12b8a7be60f48da215872265594ed948c359d8e767a29ba6e67f4d3413" exitCode=0 Sep 14 03:19:46.695200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.695028 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" event={"ID":"2e8a7b82-b28d-49b1-b806-9a53e60391df","Type":"ContainerDied","Data":"b9b1fd12b8a7be60f48da215872265594ed948c359d8e767a29ba6e67f4d3413"} Sep 14 03:19:46.695200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.695036 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" Sep 14 03:19:46.695200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.695073 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz" event={"ID":"2e8a7b82-b28d-49b1-b806-9a53e60391df","Type":"ContainerDied","Data":"8d0ffed89067a09c2d99efc204aa10b0696d8456805aaf1cb63f8cdf7e4fab05"} Sep 14 03:19:46.695200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.695095 2564 scope.go:117] "RemoveContainer" containerID="b9b1fd12b8a7be60f48da215872265594ed948c359d8e767a29ba6e67f4d3413" Sep 14 03:19:46.704713 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.704691 2564 scope.go:117] "RemoveContainer" containerID="6463dedbbc3ac9ffe97b14e850697d4c8a616e59c08189d0e6662db1a07298c2" Sep 14 03:19:46.713089 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.713072 2564 scope.go:117] "RemoveContainer" containerID="b9b1fd12b8a7be60f48da215872265594ed948c359d8e767a29ba6e67f4d3413" Sep 14 03:19:46.713416 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:19:46.713396 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9b1fd12b8a7be60f48da215872265594ed948c359d8e767a29ba6e67f4d3413\": container with ID starting with b9b1fd12b8a7be60f48da215872265594ed948c359d8e767a29ba6e67f4d3413 not found: ID does not exist" containerID="b9b1fd12b8a7be60f48da215872265594ed948c359d8e767a29ba6e67f4d3413" Sep 14 03:19:46.713461 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.713428 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b1fd12b8a7be60f48da215872265594ed948c359d8e767a29ba6e67f4d3413"} err="failed to get container status \"b9b1fd12b8a7be60f48da215872265594ed948c359d8e767a29ba6e67f4d3413\": rpc error: code = NotFound desc = could not find container \"b9b1fd12b8a7be60f48da215872265594ed948c359d8e767a29ba6e67f4d3413\": container with ID starting with b9b1fd12b8a7be60f48da215872265594ed948c359d8e767a29ba6e67f4d3413 not found: ID does not exist" Sep 14 03:19:46.713461 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.713448 2564 scope.go:117] "RemoveContainer" containerID="6463dedbbc3ac9ffe97b14e850697d4c8a616e59c08189d0e6662db1a07298c2" Sep 14 03:19:46.713776 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:19:46.713756 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6463dedbbc3ac9ffe97b14e850697d4c8a616e59c08189d0e6662db1a07298c2\": container with ID starting with 6463dedbbc3ac9ffe97b14e850697d4c8a616e59c08189d0e6662db1a07298c2 not found: ID does not exist" containerID="6463dedbbc3ac9ffe97b14e850697d4c8a616e59c08189d0e6662db1a07298c2" Sep 14 03:19:46.713813 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.713782 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6463dedbbc3ac9ffe97b14e850697d4c8a616e59c08189d0e6662db1a07298c2"} err="failed to get container status \"6463dedbbc3ac9ffe97b14e850697d4c8a616e59c08189d0e6662db1a07298c2\": rpc error: code = NotFound desc = could not find container \"6463dedbbc3ac9ffe97b14e850697d4c8a616e59c08189d0e6662db1a07298c2\": container with ID starting with 6463dedbbc3ac9ffe97b14e850697d4c8a616e59c08189d0e6662db1a07298c2 not found: ID does not exist" Sep 14 03:19:46.736232 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.736195 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz"] Sep 14 03:19:46.743390 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.743355 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-0c919e-openshift-default-7d69f7c5b8-rw8rz"] Sep 14 03:19:46.749873 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:46.749846 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e8a7b82-b28d-49b1-b806-9a53e60391df" path="/var/lib/kubelet/pods/2e8a7b82-b28d-49b1-b806-9a53e60391df/volumes" Sep 14 03:19:49.217962 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:49.217930 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:50.214257 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:50.214197 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:51.643687 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:51.643647 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:51.669463 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:51.669420 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7"] Sep 14 03:19:51.669732 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:51.669702 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7" podUID="fae11342-3dfe-4a6f-8fb8-8d7ce5a78027" containerName="payload-processing" containerID="cri-o://5f62b7fdd454747c0e6e0fc88192dc7d2c7291babb00f76a84886729ed0d9e6e" gracePeriod=30 Sep 14 03:19:51.676019 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:51.675984 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67"] Sep 14 03:19:51.676291 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:51.676210 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67" podUID="347687cf-a121-436d-85d8-271fabc52911" containerName="payload-pre-processing" containerID="cri-o://893cdcb082081b6747f6a775f5304afb56827997ff4b821d5f54e1571591cb1b" gracePeriod=30 Sep 14 03:19:51.718695 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:51.718605 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:51.795410 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:51.795372 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:51.875889 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:51.875745 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:51.953700 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:51.953666 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7" Sep 14 03:19:51.956770 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:51.956745 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67" Sep 14 03:19:52.108028 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.107988 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95zsw\" (UniqueName: \"kubernetes.io/projected/fae11342-3dfe-4a6f-8fb8-8d7ce5a78027-kube-api-access-95zsw\") pod \"fae11342-3dfe-4a6f-8fb8-8d7ce5a78027\" (UID: \"fae11342-3dfe-4a6f-8fb8-8d7ce5a78027\") " Sep 14 03:19:52.108240 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.108052 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/347687cf-a121-436d-85d8-271fabc52911-plugins-config-volume\") pod \"347687cf-a121-436d-85d8-271fabc52911\" (UID: \"347687cf-a121-436d-85d8-271fabc52911\") " Sep 14 03:19:52.108240 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.108178 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x297q\" (UniqueName: \"kubernetes.io/projected/347687cf-a121-436d-85d8-271fabc52911-kube-api-access-x297q\") pod \"347687cf-a121-436d-85d8-271fabc52911\" (UID: \"347687cf-a121-436d-85d8-271fabc52911\") " Sep 14 03:19:52.108240 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.108207 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae11342-3dfe-4a6f-8fb8-8d7ce5a78027-plugins-config-volume\") pod \"fae11342-3dfe-4a6f-8fb8-8d7ce5a78027\" (UID: \"fae11342-3dfe-4a6f-8fb8-8d7ce5a78027\") " Sep 14 03:19:52.108644 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.108604 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/347687cf-a121-436d-85d8-271fabc52911-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "347687cf-a121-436d-85d8-271fabc52911" (UID: "347687cf-a121-436d-85d8-271fabc52911"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:19:52.108789 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.108651 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae11342-3dfe-4a6f-8fb8-8d7ce5a78027-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fae11342-3dfe-4a6f-8fb8-8d7ce5a78027" (UID: "fae11342-3dfe-4a6f-8fb8-8d7ce5a78027"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:19:52.110570 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.110525 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae11342-3dfe-4a6f-8fb8-8d7ce5a78027-kube-api-access-95zsw" (OuterVolumeSpecName: "kube-api-access-95zsw") pod "fae11342-3dfe-4a6f-8fb8-8d7ce5a78027" (UID: "fae11342-3dfe-4a6f-8fb8-8d7ce5a78027"). InnerVolumeSpecName "kube-api-access-95zsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:19:52.110570 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.110549 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347687cf-a121-436d-85d8-271fabc52911-kube-api-access-x297q" (OuterVolumeSpecName: "kube-api-access-x297q") pod "347687cf-a121-436d-85d8-271fabc52911" (UID: "347687cf-a121-436d-85d8-271fabc52911"). InnerVolumeSpecName "kube-api-access-x297q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:19:52.209639 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.209541 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95zsw\" (UniqueName: \"kubernetes.io/projected/fae11342-3dfe-4a6f-8fb8-8d7ce5a78027-kube-api-access-95zsw\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:19:52.209639 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.209578 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/347687cf-a121-436d-85d8-271fabc52911-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:19:52.209639 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.209587 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x297q\" (UniqueName: \"kubernetes.io/projected/347687cf-a121-436d-85d8-271fabc52911-kube-api-access-x297q\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:19:52.209639 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.209595 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae11342-3dfe-4a6f-8fb8-8d7ce5a78027-plugins-config-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:19:52.239413 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.239347 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:52.339804 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.339770 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:52.730536 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.730501 2564 generic.go:356] "Generic (PLEG): container finished" podID="347687cf-a121-436d-85d8-271fabc52911" containerID="893cdcb082081b6747f6a775f5304afb56827997ff4b821d5f54e1571591cb1b" exitCode=0 Sep 14 03:19:52.730536 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.730536 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67" Sep 14 03:19:52.731028 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.730572 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67" event={"ID":"347687cf-a121-436d-85d8-271fabc52911","Type":"ContainerDied","Data":"893cdcb082081b6747f6a775f5304afb56827997ff4b821d5f54e1571591cb1b"} Sep 14 03:19:52.731028 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.730631 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67" event={"ID":"347687cf-a121-436d-85d8-271fabc52911","Type":"ContainerDied","Data":"6a8a46307dab5b4b9cb8e4b167afa72e56021c8fb1d715786f32ac8049647a1e"} Sep 14 03:19:52.731028 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.730660 2564 scope.go:117] "RemoveContainer" containerID="893cdcb082081b6747f6a775f5304afb56827997ff4b821d5f54e1571591cb1b" Sep 14 03:19:52.735220 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.732580 2564 generic.go:356] "Generic (PLEG): container finished" podID="fae11342-3dfe-4a6f-8fb8-8d7ce5a78027" containerID="5f62b7fdd454747c0e6e0fc88192dc7d2c7291babb00f76a84886729ed0d9e6e" exitCode=0 Sep 14 03:19:52.735220 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.732749 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7" event={"ID":"fae11342-3dfe-4a6f-8fb8-8d7ce5a78027","Type":"ContainerDied","Data":"5f62b7fdd454747c0e6e0fc88192dc7d2c7291babb00f76a84886729ed0d9e6e"} Sep 14 03:19:52.735220 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.732791 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7" event={"ID":"fae11342-3dfe-4a6f-8fb8-8d7ce5a78027","Type":"ContainerDied","Data":"19f31062b111bc29904687716ca1cc9417569152232b2ed663769924309803a6"} Sep 14 03:19:52.735220 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.732893 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7" Sep 14 03:19:52.745661 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.745638 2564 scope.go:117] "RemoveContainer" containerID="893cdcb082081b6747f6a775f5304afb56827997ff4b821d5f54e1571591cb1b" Sep 14 03:19:52.746050 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:19:52.746027 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"893cdcb082081b6747f6a775f5304afb56827997ff4b821d5f54e1571591cb1b\": container with ID starting with 893cdcb082081b6747f6a775f5304afb56827997ff4b821d5f54e1571591cb1b not found: ID does not exist" containerID="893cdcb082081b6747f6a775f5304afb56827997ff4b821d5f54e1571591cb1b" Sep 14 03:19:52.746137 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.746062 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"893cdcb082081b6747f6a775f5304afb56827997ff4b821d5f54e1571591cb1b"} err="failed to get container status \"893cdcb082081b6747f6a775f5304afb56827997ff4b821d5f54e1571591cb1b\": rpc error: code = NotFound desc = could not find container \"893cdcb082081b6747f6a775f5304afb56827997ff4b821d5f54e1571591cb1b\": container with ID starting with 893cdcb082081b6747f6a775f5304afb56827997ff4b821d5f54e1571591cb1b not found: ID does not exist" Sep 14 03:19:52.746137 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.746080 2564 scope.go:117] "RemoveContainer" containerID="5f62b7fdd454747c0e6e0fc88192dc7d2c7291babb00f76a84886729ed0d9e6e" Sep 14 03:19:52.755648 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.755624 2564 scope.go:117] "RemoveContainer" containerID="5f62b7fdd454747c0e6e0fc88192dc7d2c7291babb00f76a84886729ed0d9e6e" Sep 14 03:19:52.755995 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:19:52.755974 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f62b7fdd454747c0e6e0fc88192dc7d2c7291babb00f76a84886729ed0d9e6e\": container with ID starting with 5f62b7fdd454747c0e6e0fc88192dc7d2c7291babb00f76a84886729ed0d9e6e not found: ID does not exist" containerID="5f62b7fdd454747c0e6e0fc88192dc7d2c7291babb00f76a84886729ed0d9e6e" Sep 14 03:19:52.756070 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.756003 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f62b7fdd454747c0e6e0fc88192dc7d2c7291babb00f76a84886729ed0d9e6e"} err="failed to get container status \"5f62b7fdd454747c0e6e0fc88192dc7d2c7291babb00f76a84886729ed0d9e6e\": rpc error: code = NotFound desc = could not find container \"5f62b7fdd454747c0e6e0fc88192dc7d2c7291babb00f76a84886729ed0d9e6e\": container with ID starting with 5f62b7fdd454747c0e6e0fc88192dc7d2c7291babb00f76a84886729ed0d9e6e not found: ID does not exist" Sep 14 03:19:52.763735 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.763711 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67"] Sep 14 03:19:52.770496 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.770457 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f83c87-dbcfc978b-4bj67"] Sep 14 03:19:52.783962 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.783928 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7"] Sep 14 03:19:52.792730 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:52.792695 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f83c87-84986b885-tjvp7"] Sep 14 03:19:54.749175 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:54.749140 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="347687cf-a121-436d-85d8-271fabc52911" path="/var/lib/kubelet/pods/347687cf-a121-436d-85d8-271fabc52911/volumes" Sep 14 03:19:54.749590 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:54.749503 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fae11342-3dfe-4a6f-8fb8-8d7ce5a78027" path="/var/lib/kubelet/pods/fae11342-3dfe-4a6f-8fb8-8d7ce5a78027/volumes" Sep 14 03:19:58.549363 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:58.549326 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:58.693279 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:58.693243 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:19:58.956901 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:19:58.956852 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:20:28.889883 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:20:28.889852 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bwv6_4369115a-4032-4f5a-a3e4-c02efe665617/ovn-acl-logging/0.log" Sep 14 03:20:28.894889 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:20:28.894870 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bwv6_4369115a-4032-4f5a-a3e4-c02efe665617/ovn-acl-logging/0.log" Sep 14 03:21:24.183433 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:21:24.183402 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:21:25.751566 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:21:25.751524 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:21:29.067297 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:21:29.067264 2564 scope.go:117] "RemoveContainer" containerID="67b0c0c1c99f4a5e623dd34f3eb09a5c1e278b8c84b2b4df4827dc62c93b2ce6" Sep 14 03:21:29.075503 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:21:29.075454 2564 scope.go:117] "RemoveContainer" containerID="baf851d1b62b3192989d73d2a92095ecf9b9921c0c8eef3822a669a9cb309eca" Sep 14 03:21:34.626224 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:21:34.626184 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:21:34.727207 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:21:34.727178 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:22:09.473511 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:22:09.473456 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:22:42.226487 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:22:42.226452 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj"] Sep 14 03:22:42.228974 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:22:42.226663 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj" podUID="f29e9890-1d4d-4c19-b089-d54d991c6d61" containerName="manager" containerID="cri-o://69cb74a5dd43be18e0ac57e01c16610a6751bbb280a73f170d29a07e33652395" gracePeriod=10 Sep 14 03:22:42.465160 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:22:42.465138 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj" Sep 14 03:22:42.603161 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:22:42.603072 2564 generic.go:356] "Generic (PLEG): container finished" podID="f29e9890-1d4d-4c19-b089-d54d991c6d61" containerID="69cb74a5dd43be18e0ac57e01c16610a6751bbb280a73f170d29a07e33652395" exitCode=0 Sep 14 03:22:42.603161 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:22:42.603120 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj" Sep 14 03:22:42.603352 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:22:42.603162 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj" event={"ID":"f29e9890-1d4d-4c19-b089-d54d991c6d61","Type":"ContainerDied","Data":"69cb74a5dd43be18e0ac57e01c16610a6751bbb280a73f170d29a07e33652395"} Sep 14 03:22:42.603352 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:22:42.603218 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj" event={"ID":"f29e9890-1d4d-4c19-b089-d54d991c6d61","Type":"ContainerDied","Data":"847733ad99b6cc9c0c627ebd84130d7ec9dc5efdb5632fe54de65b895ace171b"} Sep 14 03:22:42.603352 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:22:42.603247 2564 scope.go:117] "RemoveContainer" containerID="69cb74a5dd43be18e0ac57e01c16610a6751bbb280a73f170d29a07e33652395" Sep 14 03:22:42.612645 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:22:42.612622 2564 scope.go:117] "RemoveContainer" containerID="69cb74a5dd43be18e0ac57e01c16610a6751bbb280a73f170d29a07e33652395" Sep 14 03:22:42.612926 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:22:42.612910 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69cb74a5dd43be18e0ac57e01c16610a6751bbb280a73f170d29a07e33652395\": container with ID starting with 69cb74a5dd43be18e0ac57e01c16610a6751bbb280a73f170d29a07e33652395 not found: ID does not exist" containerID="69cb74a5dd43be18e0ac57e01c16610a6751bbb280a73f170d29a07e33652395" Sep 14 03:22:42.612976 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:22:42.612936 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69cb74a5dd43be18e0ac57e01c16610a6751bbb280a73f170d29a07e33652395"} err="failed to get container status \"69cb74a5dd43be18e0ac57e01c16610a6751bbb280a73f170d29a07e33652395\": rpc error: code = NotFound desc = could not find container \"69cb74a5dd43be18e0ac57e01c16610a6751bbb280a73f170d29a07e33652395\": container with ID starting with 69cb74a5dd43be18e0ac57e01c16610a6751bbb280a73f170d29a07e33652395 not found: ID does not exist" Sep 14 03:22:42.641059 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:22:42.641035 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw4kx\" (UniqueName: \"kubernetes.io/projected/f29e9890-1d4d-4c19-b089-d54d991c6d61-kube-api-access-fw4kx\") pod \"f29e9890-1d4d-4c19-b089-d54d991c6d61\" (UID: \"f29e9890-1d4d-4c19-b089-d54d991c6d61\") " Sep 14 03:22:42.641186 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:22:42.641110 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f29e9890-1d4d-4c19-b089-d54d991c6d61-extensions-socket-volume\") pod \"f29e9890-1d4d-4c19-b089-d54d991c6d61\" (UID: \"f29e9890-1d4d-4c19-b089-d54d991c6d61\") " Sep 14 03:22:42.641441 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:22:42.641424 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29e9890-1d4d-4c19-b089-d54d991c6d61-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f29e9890-1d4d-4c19-b089-d54d991c6d61" (UID: "f29e9890-1d4d-4c19-b089-d54d991c6d61"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:22:42.643224 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:22:42.643186 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29e9890-1d4d-4c19-b089-d54d991c6d61-kube-api-access-fw4kx" (OuterVolumeSpecName: "kube-api-access-fw4kx") pod "f29e9890-1d4d-4c19-b089-d54d991c6d61" (UID: "f29e9890-1d4d-4c19-b089-d54d991c6d61"). InnerVolumeSpecName "kube-api-access-fw4kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:22:42.741842 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:22:42.741813 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f29e9890-1d4d-4c19-b089-d54d991c6d61-extensions-socket-volume\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:22:42.741842 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:22:42.741838 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw4kx\" (UniqueName: \"kubernetes.io/projected/f29e9890-1d4d-4c19-b089-d54d991c6d61-kube-api-access-fw4kx\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:22:42.920417 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:22:42.920365 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj"] Sep 14 03:22:42.925190 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:22:42.925165 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bthpj"] Sep 14 03:22:44.749421 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:22:44.749383 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f29e9890-1d4d-4c19-b089-d54d991c6d61" path="/var/lib/kubelet/pods/f29e9890-1d4d-4c19-b089-d54d991c6d61/volumes" Sep 14 03:23:29.104201 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:29.104122 2564 scope.go:117] "RemoveContainer" containerID="4d1fce75195c8a9a6c6e201be87505b29e1721a227261837505e3a8fad755411" Sep 14 03:23:29.112520 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:29.112502 2564 scope.go:117] "RemoveContainer" containerID="6579390bfd48235fa70eba9fe2927f0e580719eccece818932337744e2612a9a" Sep 14 03:23:29.120112 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:29.120092 2564 scope.go:117] "RemoveContainer" containerID="dce11b70c0cf78c2522fb5c27d0e6b2d954b0a04a320b58b9adc5dabaa2561e2" Sep 14 03:23:42.939738 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.939700 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnrq8"] Sep 14 03:23:42.940200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940022 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef0e3bc4-6eff-4176-9825-168c33aa9c03" containerName="revoke-keys" Sep 14 03:23:42.940200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940032 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef0e3bc4-6eff-4176-9825-168c33aa9c03" containerName="revoke-keys" Sep 14 03:23:42.940200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940047 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a39416c-586b-4890-9164-cea97b576dfe" containerName="payload-pre-processing" Sep 14 03:23:42.940200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940052 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a39416c-586b-4890-9164-cea97b576dfe" containerName="payload-pre-processing" Sep 14 03:23:42.940200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940062 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f29e9890-1d4d-4c19-b089-d54d991c6d61" containerName="manager" Sep 14 03:23:42.940200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940067 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f29e9890-1d4d-4c19-b089-d54d991c6d61" containerName="manager" Sep 14 03:23:42.940200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940073 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fae11342-3dfe-4a6f-8fb8-8d7ce5a78027" containerName="payload-processing" Sep 14 03:23:42.940200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940078 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fae11342-3dfe-4a6f-8fb8-8d7ce5a78027" containerName="payload-processing" Sep 14 03:23:42.940200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940089 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e8a7b82-b28d-49b1-b806-9a53e60391df" containerName="istio-proxy" Sep 14 03:23:42.940200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940094 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e8a7b82-b28d-49b1-b806-9a53e60391df" containerName="istio-proxy" Sep 14 03:23:42.940200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940107 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e8a7b82-b28d-49b1-b806-9a53e60391df" containerName="istio-proxy" Sep 14 03:23:42.940200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940112 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e8a7b82-b28d-49b1-b806-9a53e60391df" containerName="istio-proxy" Sep 14 03:23:42.940200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940118 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc" containerName="payload-processing" Sep 14 03:23:42.940200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940123 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc" containerName="payload-processing" Sep 14 03:23:42.940200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940130 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="347687cf-a121-436d-85d8-271fabc52911" containerName="payload-pre-processing" Sep 14 03:23:42.940200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940135 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="347687cf-a121-436d-85d8-271fabc52911" containerName="payload-pre-processing" Sep 14 03:23:42.940200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940141 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc6ff6da-345b-4e30-b748-723c74edaca6" containerName="istio-proxy" Sep 14 03:23:42.940200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940145 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc6ff6da-345b-4e30-b748-723c74edaca6" containerName="istio-proxy" Sep 14 03:23:42.940200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940193 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="347687cf-a121-436d-85d8-271fabc52911" containerName="payload-pre-processing" Sep 14 03:23:42.940200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940203 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f29e9890-1d4d-4c19-b089-d54d991c6d61" containerName="manager" Sep 14 03:23:42.940200 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940210 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="95c5dc92-cb1e-453d-b4f8-29cb8be0c4dc" containerName="payload-processing" Sep 14 03:23:42.940872 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940215 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a39416c-586b-4890-9164-cea97b576dfe" containerName="payload-pre-processing" Sep 14 03:23:42.940872 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940221 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fae11342-3dfe-4a6f-8fb8-8d7ce5a78027" containerName="payload-processing" Sep 14 03:23:42.940872 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940225 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e8a7b82-b28d-49b1-b806-9a53e60391df" containerName="istio-proxy" Sep 14 03:23:42.940872 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.940230 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef0e3bc4-6eff-4176-9825-168c33aa9c03" containerName="revoke-keys" Sep 14 03:23:42.943333 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.943317 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnrq8" Sep 14 03:23:42.946175 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.946156 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hd99t\"" Sep 14 03:23:42.964187 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:42.964162 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnrq8"] Sep 14 03:23:43.049801 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:43.049765 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e1a8dae-8373-47f6-b26b-3c971ed0d71f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cnrq8\" (UID: \"0e1a8dae-8373-47f6-b26b-3c971ed0d71f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnrq8" Sep 14 03:23:43.049940 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:43.049809 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8brc\" (UniqueName: \"kubernetes.io/projected/0e1a8dae-8373-47f6-b26b-3c971ed0d71f-kube-api-access-p8brc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cnrq8\" (UID: \"0e1a8dae-8373-47f6-b26b-3c971ed0d71f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnrq8" Sep 14 03:23:43.151045 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:43.151015 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e1a8dae-8373-47f6-b26b-3c971ed0d71f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cnrq8\" (UID: \"0e1a8dae-8373-47f6-b26b-3c971ed0d71f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnrq8" Sep 14 03:23:43.151149 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:43.151048 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8brc\" (UniqueName: \"kubernetes.io/projected/0e1a8dae-8373-47f6-b26b-3c971ed0d71f-kube-api-access-p8brc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cnrq8\" (UID: \"0e1a8dae-8373-47f6-b26b-3c971ed0d71f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnrq8" Sep 14 03:23:43.151374 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:43.151354 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e1a8dae-8373-47f6-b26b-3c971ed0d71f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cnrq8\" (UID: \"0e1a8dae-8373-47f6-b26b-3c971ed0d71f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnrq8" Sep 14 03:23:43.159799 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:43.159777 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8brc\" (UniqueName: \"kubernetes.io/projected/0e1a8dae-8373-47f6-b26b-3c971ed0d71f-kube-api-access-p8brc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cnrq8\" (UID: \"0e1a8dae-8373-47f6-b26b-3c971ed0d71f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnrq8" Sep 14 03:23:43.253517 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:43.253395 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnrq8" Sep 14 03:23:43.544038 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:43.544008 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnrq8"] Sep 14 03:23:43.546863 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:23:43.546838 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e1a8dae_8373_47f6_b26b_3c971ed0d71f.slice/crio-57b997bc8177b6f76957ba909016f46d7a89504be20715e95cfa56db61d01917 WatchSource:0}: Error finding container 57b997bc8177b6f76957ba909016f46d7a89504be20715e95cfa56db61d01917: Status 404 returned error can't find the container with id 57b997bc8177b6f76957ba909016f46d7a89504be20715e95cfa56db61d01917 Sep 14 03:23:43.896262 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:43.896230 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnrq8" event={"ID":"0e1a8dae-8373-47f6-b26b-3c971ed0d71f","Type":"ContainerStarted","Data":"3ff123d47f666781ee6e9486aea26f04ac14ed358684de698fb32f34f4bf8ed9"} Sep 14 03:23:43.896415 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:43.896315 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnrq8" event={"ID":"0e1a8dae-8373-47f6-b26b-3c971ed0d71f","Type":"ContainerStarted","Data":"57b997bc8177b6f76957ba909016f46d7a89504be20715e95cfa56db61d01917"} Sep 14 03:23:43.896415 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:43.896337 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnrq8" Sep 14 03:23:43.925722 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:43.925675 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnrq8" podStartSLOduration=1.925660978 podStartE2EDuration="1.925660978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:23:43.923059235 +0000 UTC m=+1995.652718363" watchObservedRunningTime="2026-09-14 03:23:43.925660978 +0000 UTC m=+1995.655320106" Sep 14 03:23:54.903296 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:23:54.903267 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cnrq8" Sep 14 03:24:24.640071 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:24.640012 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:24:29.151183 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:29.151153 2564 scope.go:117] "RemoveContainer" containerID="92e243c1c370e0ea8c498086d67e4a2b8d4ce733991e6b52305541fee36bbe85" Sep 14 03:24:38.689601 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.689569 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42"] Sep 14 03:24:38.690052 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.690026 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e8a7b82-b28d-49b1-b806-9a53e60391df" containerName="istio-proxy" Sep 14 03:24:38.693209 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.693188 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:38.696522 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.696498 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-647b49d0-kserve-self-signed-certs\"" Sep 14 03:24:38.703844 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.703824 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42"] Sep 14 03:24:38.732219 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.732191 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:24:38.804328 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.804302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnd5v\" (UniqueName: \"kubernetes.io/projected/ed2634c9-4f85-48aa-85ed-5d89010dece8-kube-api-access-gnd5v\") pod \"e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:38.804429 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.804341 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-home\") pod \"e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:38.804497 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.804431 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-tmp-dir\") pod \"e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:38.804497 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.804466 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed2634c9-4f85-48aa-85ed-5d89010dece8-tls-certs\") pod \"e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:38.804574 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.804556 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-dshm\") pod \"e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:38.804610 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.804580 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-model-cache\") pod \"e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:38.905153 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.905121 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-home\") pod \"e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:38.905281 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.905176 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-tmp-dir\") pod \"e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:38.905281 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.905195 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed2634c9-4f85-48aa-85ed-5d89010dece8-tls-certs\") pod \"e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:38.905281 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.905232 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-dshm\") pod \"e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:38.905281 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.905261 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-model-cache\") pod \"e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:38.905433 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.905318 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnd5v\" (UniqueName: \"kubernetes.io/projected/ed2634c9-4f85-48aa-85ed-5d89010dece8-kube-api-access-gnd5v\") pod \"e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:38.905557 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.905537 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-home\") pod \"e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:38.905660 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.905633 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-tmp-dir\") pod \"e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:38.905780 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.905694 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-model-cache\") pod \"e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:38.907708 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.907678 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-dshm\") pod \"e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:38.907823 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.907807 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed2634c9-4f85-48aa-85ed-5d89010dece8-tls-certs\") pod \"e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:38.913757 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:38.913740 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnd5v\" (UniqueName: \"kubernetes.io/projected/ed2634c9-4f85-48aa-85ed-5d89010dece8-kube-api-access-gnd5v\") pod \"e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:39.003199 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:39.003140 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:39.275705 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:39.275683 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42"] Sep 14 03:24:39.278912 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:24:39.278882 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded2634c9_4f85_48aa_85ed_5d89010dece8.slice/crio-1db1d327e2a6fd703e177600fd7df206c5186d10e725b41625886f904dfef0a7 WatchSource:0}: Error finding container 1db1d327e2a6fd703e177600fd7df206c5186d10e725b41625886f904dfef0a7: Status 404 returned error can't find the container with id 1db1d327e2a6fd703e177600fd7df206c5186d10e725b41625886f904dfef0a7 Sep 14 03:24:40.162154 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:40.162116 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" event={"ID":"ed2634c9-4f85-48aa-85ed-5d89010dece8","Type":"ContainerStarted","Data":"ccdb8926328bde71b899a616ba13d3f266ef7e790bb6fe7499f8ee8544a27c5d"} Sep 14 03:24:40.162154 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:40.162158 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" event={"ID":"ed2634c9-4f85-48aa-85ed-5d89010dece8","Type":"ContainerStarted","Data":"1db1d327e2a6fd703e177600fd7df206c5186d10e725b41625886f904dfef0a7"} Sep 14 03:24:40.180105 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:40.180049 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" podStartSLOduration=2.180029684 podStartE2EDuration="2.180029684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:24:40.178538503 +0000 UTC m=+2051.908197620" watchObservedRunningTime="2026-09-14 03:24:40.180029684 +0000 UTC m=+2051.909688802" Sep 14 03:24:49.004214 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:49.004168 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:49.004721 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:49.004245 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:49.016630 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:49.016608 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:49.216621 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:49.216596 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:24:52.623233 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:52.623199 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:24:54.268003 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:54.267973 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42"] Sep 14 03:24:54.288955 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:24:54.288914 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:25:05.238928 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:05.238874 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:25:05.297731 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:05.297692 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:25:09.290467 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:09.290410 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" podUID="ed2634c9-4f85-48aa-85ed-5d89010dece8" containerName="main" containerID="cri-o://ccdb8926328bde71b899a616ba13d3f266ef7e790bb6fe7499f8ee8544a27c5d" gracePeriod=45 Sep 14 03:25:09.537792 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:09.537769 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:25:09.675693 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:09.675662 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnd5v\" (UniqueName: \"kubernetes.io/projected/ed2634c9-4f85-48aa-85ed-5d89010dece8-kube-api-access-gnd5v\") pod \"ed2634c9-4f85-48aa-85ed-5d89010dece8\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " Sep 14 03:25:09.675846 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:09.675734 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-tmp-dir\") pod \"ed2634c9-4f85-48aa-85ed-5d89010dece8\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " Sep 14 03:25:09.675846 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:09.675764 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-model-cache\") pod \"ed2634c9-4f85-48aa-85ed-5d89010dece8\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " Sep 14 03:25:09.675846 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:09.675806 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-home\") pod \"ed2634c9-4f85-48aa-85ed-5d89010dece8\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " Sep 14 03:25:09.675846 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:09.675830 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed2634c9-4f85-48aa-85ed-5d89010dece8-tls-certs\") pod \"ed2634c9-4f85-48aa-85ed-5d89010dece8\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " Sep 14 03:25:09.676063 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:09.675898 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-dshm\") pod \"ed2634c9-4f85-48aa-85ed-5d89010dece8\" (UID: \"ed2634c9-4f85-48aa-85ed-5d89010dece8\") " Sep 14 03:25:09.676121 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:09.676059 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ed2634c9-4f85-48aa-85ed-5d89010dece8" (UID: "ed2634c9-4f85-48aa-85ed-5d89010dece8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:25:09.676121 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:09.676070 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-model-cache" (OuterVolumeSpecName: "model-cache") pod "ed2634c9-4f85-48aa-85ed-5d89010dece8" (UID: "ed2634c9-4f85-48aa-85ed-5d89010dece8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:25:09.676121 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:09.676100 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-home" (OuterVolumeSpecName: "home") pod "ed2634c9-4f85-48aa-85ed-5d89010dece8" (UID: "ed2634c9-4f85-48aa-85ed-5d89010dece8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:25:09.676276 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:09.676206 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-tmp-dir\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:25:09.676276 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:09.676225 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-model-cache\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:25:09.676276 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:09.676239 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-home\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:25:09.678088 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:09.678053 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-dshm" (OuterVolumeSpecName: "dshm") pod "ed2634c9-4f85-48aa-85ed-5d89010dece8" (UID: "ed2634c9-4f85-48aa-85ed-5d89010dece8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 03:25:09.678221 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:09.678118 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed2634c9-4f85-48aa-85ed-5d89010dece8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ed2634c9-4f85-48aa-85ed-5d89010dece8" (UID: "ed2634c9-4f85-48aa-85ed-5d89010dece8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 03:25:09.678221 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:09.678167 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2634c9-4f85-48aa-85ed-5d89010dece8-kube-api-access-gnd5v" (OuterVolumeSpecName: "kube-api-access-gnd5v") pod "ed2634c9-4f85-48aa-85ed-5d89010dece8" (UID: "ed2634c9-4f85-48aa-85ed-5d89010dece8"). InnerVolumeSpecName "kube-api-access-gnd5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:25:09.777544 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:09.777516 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed2634c9-4f85-48aa-85ed-5d89010dece8-tls-certs\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:25:09.777544 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:09.777541 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed2634c9-4f85-48aa-85ed-5d89010dece8-dshm\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:25:09.777544 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:09.777551 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnd5v\" (UniqueName: \"kubernetes.io/projected/ed2634c9-4f85-48aa-85ed-5d89010dece8-kube-api-access-gnd5v\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:25:10.305171 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:10.305133 2564 generic.go:356] "Generic (PLEG): container finished" podID="ed2634c9-4f85-48aa-85ed-5d89010dece8" containerID="ccdb8926328bde71b899a616ba13d3f266ef7e790bb6fe7499f8ee8544a27c5d" exitCode=0 Sep 14 03:25:10.305596 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:10.305176 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" Sep 14 03:25:10.305596 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:10.305228 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" event={"ID":"ed2634c9-4f85-48aa-85ed-5d89010dece8","Type":"ContainerDied","Data":"ccdb8926328bde71b899a616ba13d3f266ef7e790bb6fe7499f8ee8544a27c5d"} Sep 14 03:25:10.305596 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:10.305268 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42" event={"ID":"ed2634c9-4f85-48aa-85ed-5d89010dece8","Type":"ContainerDied","Data":"1db1d327e2a6fd703e177600fd7df206c5186d10e725b41625886f904dfef0a7"} Sep 14 03:25:10.305596 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:10.305286 2564 scope.go:117] "RemoveContainer" containerID="ccdb8926328bde71b899a616ba13d3f266ef7e790bb6fe7499f8ee8544a27c5d" Sep 14 03:25:10.314444 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:10.314426 2564 scope.go:117] "RemoveContainer" containerID="ccdb8926328bde71b899a616ba13d3f266ef7e790bb6fe7499f8ee8544a27c5d" Sep 14 03:25:10.314725 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:25:10.314706 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccdb8926328bde71b899a616ba13d3f266ef7e790bb6fe7499f8ee8544a27c5d\": container with ID starting with ccdb8926328bde71b899a616ba13d3f266ef7e790bb6fe7499f8ee8544a27c5d not found: ID does not exist" containerID="ccdb8926328bde71b899a616ba13d3f266ef7e790bb6fe7499f8ee8544a27c5d" Sep 14 03:25:10.314784 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:10.314734 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccdb8926328bde71b899a616ba13d3f266ef7e790bb6fe7499f8ee8544a27c5d"} err="failed to get container status \"ccdb8926328bde71b899a616ba13d3f266ef7e790bb6fe7499f8ee8544a27c5d\": rpc error: code = NotFound desc = could not find container \"ccdb8926328bde71b899a616ba13d3f266ef7e790bb6fe7499f8ee8544a27c5d\": container with ID starting with ccdb8926328bde71b899a616ba13d3f266ef7e790bb6fe7499f8ee8544a27c5d not found: ID does not exist" Sep 14 03:25:10.327660 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:10.327635 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42"] Sep 14 03:25:10.330920 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:10.330900 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-647b49d0-kserve-76b56b8975-58h42"] Sep 14 03:25:10.749835 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:10.749803 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed2634c9-4f85-48aa-85ed-5d89010dece8" path="/var/lib/kubelet/pods/ed2634c9-4f85-48aa-85ed-5d89010dece8/volumes" Sep 14 03:25:17.488519 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.488462 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9"] Sep 14 03:25:17.488992 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.488846 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed2634c9-4f85-48aa-85ed-5d89010dece8" containerName="main" Sep 14 03:25:17.488992 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.488858 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed2634c9-4f85-48aa-85ed-5d89010dece8" containerName="main" Sep 14 03:25:17.488992 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.488918 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed2634c9-4f85-48aa-85ed-5d89010dece8" containerName="main" Sep 14 03:25:17.493365 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.493342 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" Sep 14 03:25:17.496567 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.496538 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vckj5\"" Sep 14 03:25:17.496772 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.496748 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 14 03:25:17.496879 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.496829 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 03:25:17.497143 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.497118 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 14 03:25:17.497243 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.497122 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 14 03:25:17.506261 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.506239 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9"] Sep 14 03:25:17.638487 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.638451 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb8cd063-1cf8-43b0-b449-71e26bf348d0-maas-api-tls\") pod \"maas-api-59f6566bbc-knbg9\" (UID: \"cb8cd063-1cf8-43b0-b449-71e26bf348d0\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" Sep 14 03:25:17.638656 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.638524 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8cd063-1cf8-43b0-b449-71e26bf348d0-openshift-service-ca\") pod \"maas-api-59f6566bbc-knbg9\" (UID: \"cb8cd063-1cf8-43b0-b449-71e26bf348d0\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" Sep 14 03:25:17.638656 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.638547 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp56k\" (UniqueName: \"kubernetes.io/projected/cb8cd063-1cf8-43b0-b449-71e26bf348d0-kube-api-access-gp56k\") pod \"maas-api-59f6566bbc-knbg9\" (UID: \"cb8cd063-1cf8-43b0-b449-71e26bf348d0\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" Sep 14 03:25:17.638656 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.638576 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb8cd063-1cf8-43b0-b449-71e26bf348d0-metrics-certs\") pod \"maas-api-59f6566bbc-knbg9\" (UID: \"cb8cd063-1cf8-43b0-b449-71e26bf348d0\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" Sep 14 03:25:17.739669 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.739582 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb8cd063-1cf8-43b0-b449-71e26bf348d0-maas-api-tls\") pod \"maas-api-59f6566bbc-knbg9\" (UID: \"cb8cd063-1cf8-43b0-b449-71e26bf348d0\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" Sep 14 03:25:17.739669 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.739629 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8cd063-1cf8-43b0-b449-71e26bf348d0-openshift-service-ca\") pod \"maas-api-59f6566bbc-knbg9\" (UID: \"cb8cd063-1cf8-43b0-b449-71e26bf348d0\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" Sep 14 03:25:17.739669 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.739649 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp56k\" (UniqueName: \"kubernetes.io/projected/cb8cd063-1cf8-43b0-b449-71e26bf348d0-kube-api-access-gp56k\") pod \"maas-api-59f6566bbc-knbg9\" (UID: \"cb8cd063-1cf8-43b0-b449-71e26bf348d0\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" Sep 14 03:25:17.739951 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.739679 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb8cd063-1cf8-43b0-b449-71e26bf348d0-metrics-certs\") pod \"maas-api-59f6566bbc-knbg9\" (UID: \"cb8cd063-1cf8-43b0-b449-71e26bf348d0\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" Sep 14 03:25:17.740321 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.740292 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8cd063-1cf8-43b0-b449-71e26bf348d0-openshift-service-ca\") pod \"maas-api-59f6566bbc-knbg9\" (UID: \"cb8cd063-1cf8-43b0-b449-71e26bf348d0\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" Sep 14 03:25:17.742107 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.742088 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb8cd063-1cf8-43b0-b449-71e26bf348d0-maas-api-tls\") pod \"maas-api-59f6566bbc-knbg9\" (UID: \"cb8cd063-1cf8-43b0-b449-71e26bf348d0\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" Sep 14 03:25:17.742241 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.742221 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb8cd063-1cf8-43b0-b449-71e26bf348d0-metrics-certs\") pod \"maas-api-59f6566bbc-knbg9\" (UID: \"cb8cd063-1cf8-43b0-b449-71e26bf348d0\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" Sep 14 03:25:17.750447 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.750420 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp56k\" (UniqueName: \"kubernetes.io/projected/cb8cd063-1cf8-43b0-b449-71e26bf348d0-kube-api-access-gp56k\") pod \"maas-api-59f6566bbc-knbg9\" (UID: \"cb8cd063-1cf8-43b0-b449-71e26bf348d0\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" Sep 14 03:25:17.804140 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:17.804113 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" Sep 14 03:25:18.077553 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:18.077526 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9"] Sep 14 03:25:18.079195 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:25:18.079158 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8cd063_1cf8_43b0_b449_71e26bf348d0.slice/crio-4c41b9a6c434eaa305ad64b3fccf1c8806b8b38565572c9e9976997640b2d664 WatchSource:0}: Error finding container 4c41b9a6c434eaa305ad64b3fccf1c8806b8b38565572c9e9976997640b2d664: Status 404 returned error can't find the container with id 4c41b9a6c434eaa305ad64b3fccf1c8806b8b38565572c9e9976997640b2d664 Sep 14 03:25:18.080532 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:18.080515 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 03:25:18.345389 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:18.345304 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" event={"ID":"cb8cd063-1cf8-43b0-b449-71e26bf348d0","Type":"ContainerStarted","Data":"4c41b9a6c434eaa305ad64b3fccf1c8806b8b38565572c9e9976997640b2d664"} Sep 14 03:25:20.356546 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:20.356513 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" event={"ID":"cb8cd063-1cf8-43b0-b449-71e26bf348d0","Type":"ContainerStarted","Data":"e452b0653c4fe2fb7b158f22f935c76ce217f16122b804897468afc8c5c35142"} Sep 14 03:25:20.357005 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:20.356604 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" Sep 14 03:25:20.376157 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:20.376108 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" podStartSLOduration=1.7760514280000002 podStartE2EDuration="3.376092872s" totalImagesPullingTime="1.600041444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:25:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:25:20.373832796 +0000 UTC m=+2092.103491926" watchObservedRunningTime="2026-09-14 03:25:20.376092872 +0000 UTC m=+2092.105751986" Sep 14 03:25:25.469970 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:25.469932 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9"] Sep 14 03:25:25.470355 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:25.470106 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" podUID="cb8cd063-1cf8-43b0-b449-71e26bf348d0" containerName="maas-api" containerID="cri-o://e452b0653c4fe2fb7b158f22f935c76ce217f16122b804897468afc8c5c35142" gracePeriod=30 Sep 14 03:25:25.475423 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:25.475399 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" Sep 14 03:25:25.735809 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:25.735781 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" Sep 14 03:25:25.907231 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:25.907199 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb8cd063-1cf8-43b0-b449-71e26bf348d0-maas-api-tls\") pod \"cb8cd063-1cf8-43b0-b449-71e26bf348d0\" (UID: \"cb8cd063-1cf8-43b0-b449-71e26bf348d0\") " Sep 14 03:25:25.907407 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:25.907274 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8cd063-1cf8-43b0-b449-71e26bf348d0-openshift-service-ca\") pod \"cb8cd063-1cf8-43b0-b449-71e26bf348d0\" (UID: \"cb8cd063-1cf8-43b0-b449-71e26bf348d0\") " Sep 14 03:25:25.907407 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:25.907339 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb8cd063-1cf8-43b0-b449-71e26bf348d0-metrics-certs\") pod \"cb8cd063-1cf8-43b0-b449-71e26bf348d0\" (UID: \"cb8cd063-1cf8-43b0-b449-71e26bf348d0\") " Sep 14 03:25:25.907584 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:25.907447 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp56k\" (UniqueName: \"kubernetes.io/projected/cb8cd063-1cf8-43b0-b449-71e26bf348d0-kube-api-access-gp56k\") pod \"cb8cd063-1cf8-43b0-b449-71e26bf348d0\" (UID: \"cb8cd063-1cf8-43b0-b449-71e26bf348d0\") " Sep 14 03:25:25.907680 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:25.907655 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb8cd063-1cf8-43b0-b449-71e26bf348d0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cb8cd063-1cf8-43b0-b449-71e26bf348d0" (UID: "cb8cd063-1cf8-43b0-b449-71e26bf348d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 03:25:25.909679 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:25.909655 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb8cd063-1cf8-43b0-b449-71e26bf348d0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cb8cd063-1cf8-43b0-b449-71e26bf348d0" (UID: "cb8cd063-1cf8-43b0-b449-71e26bf348d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 03:25:25.909775 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:25.909673 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb8cd063-1cf8-43b0-b449-71e26bf348d0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cb8cd063-1cf8-43b0-b449-71e26bf348d0" (UID: "cb8cd063-1cf8-43b0-b449-71e26bf348d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 03:25:25.909775 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:25.909673 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8cd063-1cf8-43b0-b449-71e26bf348d0-kube-api-access-gp56k" (OuterVolumeSpecName: "kube-api-access-gp56k") pod "cb8cd063-1cf8-43b0-b449-71e26bf348d0" (UID: "cb8cd063-1cf8-43b0-b449-71e26bf348d0"). InnerVolumeSpecName "kube-api-access-gp56k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 03:25:26.008079 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:26.008026 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb8cd063-1cf8-43b0-b449-71e26bf348d0-maas-api-tls\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:25:26.008079 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:26.008047 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8cd063-1cf8-43b0-b449-71e26bf348d0-openshift-service-ca\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:25:26.008079 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:26.008058 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb8cd063-1cf8-43b0-b449-71e26bf348d0-metrics-certs\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:25:26.008079 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:26.008067 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp56k\" (UniqueName: \"kubernetes.io/projected/cb8cd063-1cf8-43b0-b449-71e26bf348d0-kube-api-access-gp56k\") on node \"ip-10-0-129-88.ec2.internal\" DevicePath \"\"" Sep 14 03:25:26.388632 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:26.388546 2564 generic.go:356] "Generic (PLEG): container finished" podID="cb8cd063-1cf8-43b0-b449-71e26bf348d0" containerID="e452b0653c4fe2fb7b158f22f935c76ce217f16122b804897468afc8c5c35142" exitCode=0 Sep 14 03:25:26.388632 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:26.388597 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" Sep 14 03:25:26.388859 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:26.388598 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" event={"ID":"cb8cd063-1cf8-43b0-b449-71e26bf348d0","Type":"ContainerDied","Data":"e452b0653c4fe2fb7b158f22f935c76ce217f16122b804897468afc8c5c35142"} Sep 14 03:25:26.388859 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:26.388717 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9" event={"ID":"cb8cd063-1cf8-43b0-b449-71e26bf348d0","Type":"ContainerDied","Data":"4c41b9a6c434eaa305ad64b3fccf1c8806b8b38565572c9e9976997640b2d664"} Sep 14 03:25:26.388859 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:26.388750 2564 scope.go:117] "RemoveContainer" containerID="e452b0653c4fe2fb7b158f22f935c76ce217f16122b804897468afc8c5c35142" Sep 14 03:25:26.398825 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:26.398806 2564 scope.go:117] "RemoveContainer" containerID="e452b0653c4fe2fb7b158f22f935c76ce217f16122b804897468afc8c5c35142" Sep 14 03:25:26.399113 ip-10-0-129-88 kubenswrapper[2564]: E0914 03:25:26.399095 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e452b0653c4fe2fb7b158f22f935c76ce217f16122b804897468afc8c5c35142\": container with ID starting with e452b0653c4fe2fb7b158f22f935c76ce217f16122b804897468afc8c5c35142 not found: ID does not exist" containerID="e452b0653c4fe2fb7b158f22f935c76ce217f16122b804897468afc8c5c35142" Sep 14 03:25:26.399188 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:26.399121 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e452b0653c4fe2fb7b158f22f935c76ce217f16122b804897468afc8c5c35142"} err="failed to get container status \"e452b0653c4fe2fb7b158f22f935c76ce217f16122b804897468afc8c5c35142\": rpc error: code = NotFound desc = could not find container \"e452b0653c4fe2fb7b158f22f935c76ce217f16122b804897468afc8c5c35142\": container with ID starting with e452b0653c4fe2fb7b158f22f935c76ce217f16122b804897468afc8c5c35142 not found: ID does not exist" Sep 14 03:25:26.412776 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:26.412755 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9"] Sep 14 03:25:26.420221 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:26.420198 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-knbg9"] Sep 14 03:25:26.751055 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:26.751024 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb8cd063-1cf8-43b0-b449-71e26bf348d0" path="/var/lib/kubelet/pods/cb8cd063-1cf8-43b0-b449-71e26bf348d0/volumes" Sep 14 03:25:28.925389 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:28.925362 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bwv6_4369115a-4032-4f5a-a3e4-c02efe665617/ovn-acl-logging/0.log" Sep 14 03:25:28.930643 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:28.930623 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5bwv6_4369115a-4032-4f5a-a3e4-c02efe665617/ovn-acl-logging/0.log" Sep 14 03:25:29.169869 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:29.169842 2564 scope.go:117] "RemoveContainer" containerID="e3ea54ff51fb5ca13c0489bed6dad7c3559d107fa2479656814f78ee7ab5fc71" Sep 14 03:25:47.546987 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:47.546950 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9xgtv/must-gather-wtjrg"] Sep 14 03:25:47.547381 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:47.547301 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb8cd063-1cf8-43b0-b449-71e26bf348d0" containerName="maas-api" Sep 14 03:25:47.547381 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:47.547312 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb8cd063-1cf8-43b0-b449-71e26bf348d0" containerName="maas-api" Sep 14 03:25:47.547381 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:47.547367 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb8cd063-1cf8-43b0-b449-71e26bf348d0" containerName="maas-api" Sep 14 03:25:47.550432 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:47.550415 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xgtv/must-gather-wtjrg" Sep 14 03:25:47.553056 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:47.553025 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9xgtv\"/\"default-dockercfg-dfvsv\"" Sep 14 03:25:47.553329 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:47.553307 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9xgtv\"/\"kube-root-ca.crt\"" Sep 14 03:25:47.553465 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:47.553447 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9xgtv\"/\"openshift-service-ca.crt\"" Sep 14 03:25:47.564025 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:47.564000 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xgtv/must-gather-wtjrg"] Sep 14 03:25:47.571654 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:47.571628 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:25:47.583464 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:47.583443 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smt86\" (UniqueName: \"kubernetes.io/projected/b23fd64f-ff2d-4747-9751-c36eeacec90d-kube-api-access-smt86\") pod \"must-gather-wtjrg\" (UID: \"b23fd64f-ff2d-4747-9751-c36eeacec90d\") " pod="openshift-must-gather-9xgtv/must-gather-wtjrg" Sep 14 03:25:47.583590 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:47.583504 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b23fd64f-ff2d-4747-9751-c36eeacec90d-must-gather-output\") pod \"must-gather-wtjrg\" (UID: \"b23fd64f-ff2d-4747-9751-c36eeacec90d\") " pod="openshift-must-gather-9xgtv/must-gather-wtjrg" Sep 14 03:25:47.684759 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:47.684726 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smt86\" (UniqueName: \"kubernetes.io/projected/b23fd64f-ff2d-4747-9751-c36eeacec90d-kube-api-access-smt86\") pod \"must-gather-wtjrg\" (UID: \"b23fd64f-ff2d-4747-9751-c36eeacec90d\") " pod="openshift-must-gather-9xgtv/must-gather-wtjrg" Sep 14 03:25:47.684934 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:47.684781 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b23fd64f-ff2d-4747-9751-c36eeacec90d-must-gather-output\") pod \"must-gather-wtjrg\" (UID: \"b23fd64f-ff2d-4747-9751-c36eeacec90d\") " pod="openshift-must-gather-9xgtv/must-gather-wtjrg" Sep 14 03:25:47.685194 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:47.685172 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b23fd64f-ff2d-4747-9751-c36eeacec90d-must-gather-output\") pod \"must-gather-wtjrg\" (UID: \"b23fd64f-ff2d-4747-9751-c36eeacec90d\") " pod="openshift-must-gather-9xgtv/must-gather-wtjrg" Sep 14 03:25:47.693517 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:47.693486 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" limit=1024 Sep 14 03:25:47.699450 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:47.699427 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smt86\" (UniqueName: \"kubernetes.io/projected/b23fd64f-ff2d-4747-9751-c36eeacec90d-kube-api-access-smt86\") pod \"must-gather-wtjrg\" (UID: \"b23fd64f-ff2d-4747-9751-c36eeacec90d\") " pod="openshift-must-gather-9xgtv/must-gather-wtjrg" Sep 14 03:25:47.860016 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:47.859958 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xgtv/must-gather-wtjrg" Sep 14 03:25:48.133317 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:48.133292 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xgtv/must-gather-wtjrg"] Sep 14 03:25:48.135026 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:25:48.134997 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb23fd64f_ff2d_4747_9751_c36eeacec90d.slice/crio-0b713452bd8ef5d8ebe15c9ce48e355b38da2a41f803ef40f510030496be22fa WatchSource:0}: Error finding container 0b713452bd8ef5d8ebe15c9ce48e355b38da2a41f803ef40f510030496be22fa: Status 404 returned error can't find the container with id 0b713452bd8ef5d8ebe15c9ce48e355b38da2a41f803ef40f510030496be22fa Sep 14 03:25:48.495774 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:48.495735 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xgtv/must-gather-wtjrg" event={"ID":"b23fd64f-ff2d-4747-9751-c36eeacec90d","Type":"ContainerStarted","Data":"0b713452bd8ef5d8ebe15c9ce48e355b38da2a41f803ef40f510030496be22fa"} Sep 14 03:25:49.502846 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:49.502807 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xgtv/must-gather-wtjrg" event={"ID":"b23fd64f-ff2d-4747-9751-c36eeacec90d","Type":"ContainerStarted","Data":"2f93df471978d6ecc4b1845c2cb3a26658b5cb9d2a5ac07ab733e42894c9e3b0"} Sep 14 03:25:49.503322 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:49.502874 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xgtv/must-gather-wtjrg" event={"ID":"b23fd64f-ff2d-4747-9751-c36eeacec90d","Type":"ContainerStarted","Data":"17b87b727d0955afb1b30ec122d1558e69f17816523c0b5b8b262f79ac56479a"} Sep 14 03:25:49.523642 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:49.523248 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9xgtv/must-gather-wtjrg" podStartSLOduration=1.7923843449999999 podStartE2EDuration="2.52322808s" totalImagesPullingTime="730.843735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 03:25:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:25:49.520955385 +0000 UTC m=+2121.250614502" watchObservedRunningTime="2026-09-14 03:25:49.52322808 +0000 UTC m=+2121.252887198" Sep 14 03:25:50.687324 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:50.687290 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mq8rq_bc313df4-a60b-4df9-8d21-bb1141d7df64/global-pull-secret-syncer/0.log" Sep 14 03:25:50.794629 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:50.794599 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6sn84_48bf1d45-d3d3-4bde-8f44-688b868762d3/konnectivity-agent/0.log" Sep 14 03:25:50.887843 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:50.887813 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-88.ec2.internal_edcf5b83cf757b447b27e9d1ae1ebd2b/haproxy/0.log" Sep 14 03:25:55.503907 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:55.503872 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65dd4bc4dc-x28vm_9e5d1c0d-0a82-4c66-8013-2d1648113354/authorino/0.log" Sep 14 03:25:55.636863 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:55.636834 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-cnrq8_0e1a8dae-8373-47f6-b26b-3c971ed0d71f/manager/0.log" Sep 14 03:25:57.168678 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:57.168646 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2c5693c3-3949-4015-9a7b-f5451574e59c/alertmanager/0.log" Sep 14 03:25:57.194034 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:57.193975 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2c5693c3-3949-4015-9a7b-f5451574e59c/config-reloader/0.log" Sep 14 03:25:57.218818 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:57.218791 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2c5693c3-3949-4015-9a7b-f5451574e59c/kube-rbac-proxy-web/0.log" Sep 14 03:25:57.244498 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:57.244446 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2c5693c3-3949-4015-9a7b-f5451574e59c/kube-rbac-proxy/0.log" Sep 14 03:25:57.269048 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:57.269009 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2c5693c3-3949-4015-9a7b-f5451574e59c/kube-rbac-proxy-metric/0.log" Sep 14 03:25:57.294517 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:57.294487 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2c5693c3-3949-4015-9a7b-f5451574e59c/prom-label-proxy/0.log" Sep 14 03:25:57.324875 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:57.324780 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2c5693c3-3949-4015-9a7b-f5451574e59c/init-config-reloader/0.log" Sep 14 03:25:57.409852 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:57.409824 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-w24sd_049d80ec-0298-4f01-a744-5bfcd8ea6bcf/kube-state-metrics/0.log" Sep 14 03:25:57.434375 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:57.434297 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-w24sd_049d80ec-0298-4f01-a744-5bfcd8ea6bcf/kube-rbac-proxy-main/0.log" Sep 14 03:25:57.457250 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:57.457224 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-w24sd_049d80ec-0298-4f01-a744-5bfcd8ea6bcf/kube-rbac-proxy-self/0.log" Sep 14 03:25:57.495859 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:57.495830 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-688bd89fc-t949d_c02c878c-a03b-42ed-b288-6b044226cee0/metrics-server/0.log" Sep 14 03:25:57.735216 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:57.735142 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v8xq9_42ab45b2-5436-4392-b8a1-000f53af76df/node-exporter/0.log" Sep 14 03:25:57.771000 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:57.770977 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v8xq9_42ab45b2-5436-4392-b8a1-000f53af76df/kube-rbac-proxy/0.log" Sep 14 03:25:57.802571 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:57.802377 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-v8xq9_42ab45b2-5436-4392-b8a1-000f53af76df/init-textfile/0.log" Sep 14 03:25:57.953504 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:57.952818 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0bfc948-e0b8-47d2-9e84-95f519f3411a/prometheus/0.log" Sep 14 03:25:57.992305 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:57.992224 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0bfc948-e0b8-47d2-9e84-95f519f3411a/config-reloader/0.log" Sep 14 03:25:58.023064 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:58.023035 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0bfc948-e0b8-47d2-9e84-95f519f3411a/thanos-sidecar/0.log" Sep 14 03:25:58.049627 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:58.049600 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0bfc948-e0b8-47d2-9e84-95f519f3411a/kube-rbac-proxy-web/0.log" Sep 14 03:25:58.080973 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:58.080935 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0bfc948-e0b8-47d2-9e84-95f519f3411a/kube-rbac-proxy/0.log" Sep 14 03:25:58.107800 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:58.107763 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0bfc948-e0b8-47d2-9e84-95f519f3411a/kube-rbac-proxy-thanos/0.log" Sep 14 03:25:58.138118 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:58.138084 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e0bfc948-e0b8-47d2-9e84-95f519f3411a/init-config-reloader/0.log" Sep 14 03:25:58.288720 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:58.288650 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56b4cc6586-zhfrt_1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b/telemeter-client/0.log" Sep 14 03:25:58.315233 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:58.315198 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56b4cc6586-zhfrt_1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b/reload/0.log" Sep 14 03:25:58.347288 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:25:58.347256 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56b4cc6586-zhfrt_1c2ae1ea-bde9-4ff8-8f4c-0e28e39ff11b/kube-rbac-proxy/0.log" Sep 14 03:26:00.877857 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:00.877818 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn"] Sep 14 03:26:00.885766 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:00.885734 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn" Sep 14 03:26:00.890240 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:00.890210 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn"] Sep 14 03:26:01.046392 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:01.046360 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c46cf39-39cd-4196-90c6-e6e41682918f-sys\") pod \"perf-node-gather-daemonset-hvhqn\" (UID: \"9c46cf39-39cd-4196-90c6-e6e41682918f\") " pod="openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn" Sep 14 03:26:01.046612 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:01.046422 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9c46cf39-39cd-4196-90c6-e6e41682918f-proc\") pod \"perf-node-gather-daemonset-hvhqn\" (UID: \"9c46cf39-39cd-4196-90c6-e6e41682918f\") " pod="openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn" Sep 14 03:26:01.046612 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:01.046492 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c46cf39-39cd-4196-90c6-e6e41682918f-lib-modules\") pod \"perf-node-gather-daemonset-hvhqn\" (UID: \"9c46cf39-39cd-4196-90c6-e6e41682918f\") " pod="openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn" Sep 14 03:26:01.046612 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:01.046540 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw5tm\" (UniqueName: \"kubernetes.io/projected/9c46cf39-39cd-4196-90c6-e6e41682918f-kube-api-access-hw5tm\") pod \"perf-node-gather-daemonset-hvhqn\" (UID: \"9c46cf39-39cd-4196-90c6-e6e41682918f\") " pod="openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn" Sep 14 03:26:01.046612 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:01.046570 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9c46cf39-39cd-4196-90c6-e6e41682918f-podres\") pod \"perf-node-gather-daemonset-hvhqn\" (UID: \"9c46cf39-39cd-4196-90c6-e6e41682918f\") " pod="openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn" Sep 14 03:26:01.147865 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:01.147828 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c46cf39-39cd-4196-90c6-e6e41682918f-sys\") pod \"perf-node-gather-daemonset-hvhqn\" (UID: \"9c46cf39-39cd-4196-90c6-e6e41682918f\") " pod="openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn" Sep 14 03:26:01.148297 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:01.148275 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9c46cf39-39cd-4196-90c6-e6e41682918f-proc\") pod \"perf-node-gather-daemonset-hvhqn\" (UID: \"9c46cf39-39cd-4196-90c6-e6e41682918f\") " pod="openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn" Sep 14 03:26:01.148574 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:01.148555 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c46cf39-39cd-4196-90c6-e6e41682918f-lib-modules\") pod \"perf-node-gather-daemonset-hvhqn\" (UID: \"9c46cf39-39cd-4196-90c6-e6e41682918f\") " pod="openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn" Sep 14 03:26:01.148887 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:01.148868 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw5tm\" (UniqueName: \"kubernetes.io/projected/9c46cf39-39cd-4196-90c6-e6e41682918f-kube-api-access-hw5tm\") pod \"perf-node-gather-daemonset-hvhqn\" (UID: \"9c46cf39-39cd-4196-90c6-e6e41682918f\") " pod="openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn" Sep 14 03:26:01.149404 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:01.149385 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9c46cf39-39cd-4196-90c6-e6e41682918f-podres\") pod \"perf-node-gather-daemonset-hvhqn\" (UID: \"9c46cf39-39cd-4196-90c6-e6e41682918f\") " pod="openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn" Sep 14 03:26:01.149732 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:01.148470 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9c46cf39-39cd-4196-90c6-e6e41682918f-proc\") pod \"perf-node-gather-daemonset-hvhqn\" (UID: \"9c46cf39-39cd-4196-90c6-e6e41682918f\") " pod="openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn" Sep 14 03:26:01.149874 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:01.148165 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9c46cf39-39cd-4196-90c6-e6e41682918f-sys\") pod \"perf-node-gather-daemonset-hvhqn\" (UID: \"9c46cf39-39cd-4196-90c6-e6e41682918f\") " pod="openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn" Sep 14 03:26:01.150001 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:01.149633 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9c46cf39-39cd-4196-90c6-e6e41682918f-podres\") pod \"perf-node-gather-daemonset-hvhqn\" (UID: \"9c46cf39-39cd-4196-90c6-e6e41682918f\") " pod="openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn" Sep 14 03:26:01.150100 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:01.148806 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9c46cf39-39cd-4196-90c6-e6e41682918f-lib-modules\") pod \"perf-node-gather-daemonset-hvhqn\" (UID: \"9c46cf39-39cd-4196-90c6-e6e41682918f\") " pod="openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn" Sep 14 03:26:01.158082 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:01.158051 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw5tm\" (UniqueName: \"kubernetes.io/projected/9c46cf39-39cd-4196-90c6-e6e41682918f-kube-api-access-hw5tm\") pod \"perf-node-gather-daemonset-hvhqn\" (UID: \"9c46cf39-39cd-4196-90c6-e6e41682918f\") " pod="openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn" Sep 14 03:26:01.201260 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:01.201217 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn" Sep 14 03:26:01.266256 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:01.266196 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-dqfwl_06abfff7-30d4-4770-994a-ee2fd0215cd5/download-server/0.log" Sep 14 03:26:01.485340 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:01.485137 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn"] Sep 14 03:26:01.488297 ip-10-0-129-88 kubenswrapper[2564]: W0914 03:26:01.488262 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9c46cf39_39cd_4196_90c6_e6e41682918f.slice/crio-bf334fc499fedfb09c368db972a32bd0816dd19bd8350d52cff7fe4a126ffc9d WatchSource:0}: Error finding container bf334fc499fedfb09c368db972a32bd0816dd19bd8350d52cff7fe4a126ffc9d: Status 404 returned error can't find the container with id bf334fc499fedfb09c368db972a32bd0816dd19bd8350d52cff7fe4a126ffc9d Sep 14 03:26:01.589621 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:01.589582 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn" event={"ID":"9c46cf39-39cd-4196-90c6-e6e41682918f","Type":"ContainerStarted","Data":"bf334fc499fedfb09c368db972a32bd0816dd19bd8350d52cff7fe4a126ffc9d"} Sep 14 03:26:02.597882 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:02.597848 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn" event={"ID":"9c46cf39-39cd-4196-90c6-e6e41682918f","Type":"ContainerStarted","Data":"ceed81ddc9a4ba42a001df5e4ce955c3efc78af1509b9b3545dd6e384e194b88"} Sep 14 03:26:02.598289 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:02.597977 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn" Sep 14 03:26:02.633450 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:02.633401 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn" podStartSLOduration=2.633381832 podStartE2EDuration="2.633381832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 03:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 03:26:02.629834473 +0000 UTC m=+2134.359493589" watchObservedRunningTime="2026-09-14 03:26:02.633381832 +0000 UTC m=+2134.363040949" Sep 14 03:26:03.278703 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:03.278670 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-znkd5_47b16be3-755a-4a89-a2ef-830c72498f05/dns/0.log" Sep 14 03:26:03.305049 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:03.305023 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-znkd5_47b16be3-755a-4a89-a2ef-830c72498f05/kube-rbac-proxy/0.log" Sep 14 03:26:03.394626 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:03.394593 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xgj42_06330f4e-f78f-49f6-8c53-9565d0364232/dns-node-resolver/0.log" Sep 14 03:26:04.070102 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:04.070070 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tjwgv_58a26876-794d-43a3-9b28-d8375839bbcf/node-ca/0.log" Sep 14 03:26:05.664939 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:05.664902 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66f955bb57-bjdhw_800ff9e9-2373-4e6b-aafb-cfe2a0fb7126/kube-auth-proxy/0.log" Sep 14 03:26:06.684990 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:06.684961 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8px75_1b6e2fea-3457-4b62-ba40-0254ed00fb1e/serve-healthcheck-canary/0.log" Sep 14 03:26:07.327793 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:07.327761 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kfljj_83942481-0d5f-46ca-b957-7a5ba2066b7d/kube-rbac-proxy/0.log" Sep 14 03:26:07.352847 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:07.352817 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kfljj_83942481-0d5f-46ca-b957-7a5ba2066b7d/exporter/0.log" Sep 14 03:26:07.404545 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:07.404510 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kfljj_83942481-0d5f-46ca-b957-7a5ba2066b7d/extractor/0.log" Sep 14 03:26:08.613971 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:08.613927 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9xgtv/perf-node-gather-daemonset-hvhqn" Sep 14 03:26:09.937823 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:09.937745 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-76dfc9b5ff-rgrbx_913b62ee-5883-4444-8e49-00e24996683a/manager/0.log" Sep 14 03:26:10.542929 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:10.542899 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f647699c6-5r6gz_3f4924af-4868-4817-b105-23e45657a58c/manager/0.log" Sep 14 03:26:10.569820 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:10.569790 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f647699c6-5r6gz_3f4924af-4868-4817-b105-23e45657a58c/copy-manifests/0.log" Sep 14 03:26:11.621308 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:11.621276 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66f955bb57-bjdhw_800ff9e9-2373-4e6b-aafb-cfe2a0fb7126/kube-auth-proxy/0.log" Sep 14 03:26:11.924338 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:11.924298 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5647bd797c-fzbg8_3201c1d8-e6fc-43bd-8b88-4c5827d6b74d/manager/0.log" Sep 14 03:26:11.986745 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:11.986710 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-l5sdn_fc9f26cd-12ca-42a4-a680-93e4b1878b86/openshift-lws-operator/0.log" Sep 14 03:26:13.788804 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:13.788780 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66f955bb57-bjdhw_800ff9e9-2373-4e6b-aafb-cfe2a0fb7126/kube-auth-proxy/0.log" Sep 14 03:26:17.056686 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:17.056650 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-dtzk7_e1f8f137-bf5f-4e2d-8833-00b4d255aa9c/migrator/0.log" Sep 14 03:26:17.086628 ip-10-0-129-88 kubenswrapper[2564]: I0914 03:26:17.086599 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-dtzk7_e1f8f137-bf5f-4e2d-8833-00b4d255aa9c/graceful-termination/0.log"