Sep 11 14:14:49.256751 ip-10-0-135-17 systemd[1]: Starting Kubernetes Kubelet... Sep 11 14:14:49.712396 ip-10-0-135-17 kubenswrapper[2553]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 14:14:49.712396 ip-10-0-135-17 kubenswrapper[2553]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 14:14:49.712396 ip-10-0-135-17 kubenswrapper[2553]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 14:14:49.712396 ip-10-0-135-17 kubenswrapper[2553]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 14:14:49.712396 ip-10-0-135-17 kubenswrapper[2553]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 14:14:49.714774 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.714690 2553 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 14:14:49.718772 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718757 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:14:49.718772 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718772 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:14:49.718838 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718777 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:14:49.718838 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718780 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:14:49.718838 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718784 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:14:49.718838 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718787 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:14:49.718838 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718789 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:14:49.718838 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718793 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:14:49.718838 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718796 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:14:49.718838 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718798 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:14:49.718838 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718801 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:14:49.718838 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718804 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:14:49.718838 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718807 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:14:49.718838 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718810 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:14:49.718838 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718812 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:14:49.718838 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718819 2553 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:14:49.718838 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718822 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:14:49.718838 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718825 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:14:49.718838 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718828 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:14:49.718838 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718830 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:14:49.718838 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718833 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:14:49.719309 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718835 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:14:49.719309 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718838 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:14:49.719309 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718840 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:14:49.719309 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718843 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:14:49.719309 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718846 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:14:49.719309 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718849 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:14:49.719309 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718852 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:14:49.719309 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718855 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:14:49.719309 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718857 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:14:49.719309 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718860 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:14:49.719309 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718862 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:14:49.719309 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718865 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:14:49.719309 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718867 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:14:49.719309 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718870 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:14:49.719309 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718872 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:14:49.719309 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718875 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:14:49.719309 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718877 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:14:49.719309 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718880 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:14:49.719309 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718882 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:14:49.719309 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718885 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:14:49.719954 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718887 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:14:49.719954 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718891 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:14:49.719954 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718895 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:14:49.719954 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718898 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:14:49.719954 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718900 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:14:49.719954 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718903 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:14:49.719954 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718905 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:14:49.719954 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718909 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:14:49.719954 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718912 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:14:49.719954 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718914 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:14:49.719954 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718917 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:14:49.719954 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718919 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:14:49.719954 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718922 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:14:49.719954 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718925 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:14:49.719954 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718928 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:14:49.719954 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718931 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:14:49.719954 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718934 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:14:49.719954 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718936 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:14:49.719954 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718939 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:14:49.719954 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718941 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:14:49.720665 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718944 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:14:49.720665 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718946 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:14:49.720665 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718949 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:14:49.720665 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718953 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:14:49.720665 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718956 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:14:49.720665 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718960 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:14:49.720665 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718964 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:14:49.720665 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718967 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:14:49.720665 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718970 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:14:49.720665 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718973 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:14:49.720665 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718976 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:14:49.720665 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718978 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:14:49.720665 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718981 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:14:49.720665 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718983 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:14:49.720665 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718986 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:14:49.720665 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718989 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:14:49.720665 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718992 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:14:49.720665 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718994 2553 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:14:49.720665 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.718997 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:14:49.721129 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.719000 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:14:49.721129 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.719003 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:14:49.721129 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.719006 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:14:49.721129 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.719008 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:14:49.721129 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.719011 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:14:49.721129 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.719013 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:14:49.721129 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.719016 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:14:49.721129 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.719019 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:14:49.721129 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.719021 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:14:49.721129 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.719025 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:14:49.721129 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.719028 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:14:49.721129 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.719030 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:14:49.721129 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.719033 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:14:49.721129 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.719035 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:14:49.721129 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.719038 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:14:49.721129 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.719040 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:14:49.721129 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.719043 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:14:49.721129 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.719046 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:14:49.721129 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.719049 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:14:49.721129 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720289 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:14:49.721630 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720303 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:14:49.721630 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720309 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:14:49.721630 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720314 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:14:49.721630 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720319 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:14:49.721630 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720323 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:14:49.721630 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720327 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:14:49.721630 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720331 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:14:49.721630 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720335 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:14:49.721630 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720341 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:14:49.721630 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720346 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:14:49.721630 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720358 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:14:49.721630 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720362 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:14:49.721630 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720366 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:14:49.721630 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720371 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:14:49.721630 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720377 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:14:49.721630 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720381 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:14:49.721630 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720386 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:14:49.721630 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720391 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:14:49.721630 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720395 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:14:49.721630 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720400 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:14:49.722226 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720405 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:14:49.722226 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720409 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:14:49.722226 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720437 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:14:49.722226 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720442 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:14:49.722226 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720446 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:14:49.722226 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720449 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:14:49.722226 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720452 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:14:49.722226 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720455 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:14:49.722226 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720458 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:14:49.722226 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720461 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:14:49.722226 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720465 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:14:49.722226 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720470 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:14:49.722226 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720475 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:14:49.722226 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720484 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:14:49.722226 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720487 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:14:49.722226 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720490 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:14:49.722226 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720495 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:14:49.722226 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720499 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:14:49.722226 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720506 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:14:49.722226 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720512 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:14:49.722742 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720517 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:14:49.722742 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720530 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:14:49.722742 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720555 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:14:49.722742 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720658 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:14:49.722742 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720662 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:14:49.722742 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720667 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:14:49.722742 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720672 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:14:49.722742 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720677 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:14:49.722742 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720681 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:14:49.722742 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720685 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:14:49.722742 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720690 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:14:49.722742 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720694 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:14:49.722742 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720700 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:14:49.722742 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720705 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:14:49.722742 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720710 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:14:49.722742 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720714 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:14:49.722742 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720717 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:14:49.722742 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720722 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:14:49.722742 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720726 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:14:49.723209 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720730 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:14:49.723209 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720734 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:14:49.723209 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720741 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:14:49.723209 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720745 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:14:49.723209 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720749 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:14:49.723209 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720753 2553 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:14:49.723209 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720757 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:14:49.723209 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720762 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:14:49.723209 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720766 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:14:49.723209 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720770 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:14:49.723209 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720775 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:14:49.723209 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720782 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:14:49.723209 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720787 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:14:49.723209 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720791 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:14:49.723209 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720796 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:14:49.723209 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720800 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:14:49.723209 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720804 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:14:49.723209 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720808 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:14:49.723209 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720812 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:14:49.723209 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720816 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:14:49.723733 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720822 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:14:49.723733 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720826 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:14:49.723733 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720831 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:14:49.723733 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720834 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:14:49.723733 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720837 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:14:49.723733 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720839 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:14:49.723733 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720842 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:14:49.723733 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720845 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:14:49.723733 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720847 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:14:49.723733 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720850 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:14:49.723733 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720853 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:14:49.723733 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720855 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:14:49.723733 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720858 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:14:49.723733 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720860 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:14:49.723733 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720863 2553 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:14:49.723733 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720865 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:14:49.723733 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720868 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:14:49.723733 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720870 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:14:49.723733 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.720873 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:14:49.723733 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721236 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:14:49.724221 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721241 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:14:49.724221 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721244 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:14:49.724221 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721246 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:14:49.724221 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721249 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:14:49.724221 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721251 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:14:49.724221 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721256 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:14:49.724221 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721260 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:14:49.724221 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721263 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:14:49.724221 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721266 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:14:49.724221 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721269 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:14:49.724221 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721271 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:14:49.724221 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721274 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:14:49.724221 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721277 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:14:49.724221 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721279 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:14:49.724221 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721282 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:14:49.724221 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721286 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:14:49.724221 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721288 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:14:49.724221 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721291 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:14:49.724221 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721294 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:14:49.724221 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721296 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:14:49.724757 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721299 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:14:49.724757 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721301 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:14:49.724757 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721304 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:14:49.724757 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721306 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:14:49.724757 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721308 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:14:49.724757 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721311 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:14:49.724757 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721313 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:14:49.724757 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721316 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:14:49.724757 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721319 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:14:49.724757 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721321 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:14:49.724757 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721324 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:14:49.724757 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721326 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:14:49.724757 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721329 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:14:49.724757 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721331 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:14:49.724757 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721334 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:14:49.724757 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721336 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:14:49.724757 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721339 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:14:49.724757 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721341 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:14:49.724757 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721346 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:14:49.724757 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721349 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:14:49.725249 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721352 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:14:49.725249 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721354 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:14:49.725249 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721357 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:14:49.725249 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721359 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:14:49.725249 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721362 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:14:49.725249 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721365 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:14:49.725249 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721367 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:14:49.725249 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721370 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:14:49.725249 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721374 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:14:49.725249 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721376 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:14:49.725249 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721378 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:14:49.725249 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721381 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:14:49.725249 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721383 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:14:49.725249 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721386 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:14:49.725249 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721388 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:14:49.725249 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721391 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:14:49.725249 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721393 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:14:49.725249 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721396 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:14:49.725249 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721398 2553 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:14:49.725249 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721401 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:14:49.725779 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721403 2553 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:14:49.725779 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721407 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:14:49.725779 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721427 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:14:49.725779 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721432 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:14:49.725779 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721438 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:14:49.725779 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721442 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:14:49.725779 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721445 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:14:49.725779 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721448 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:14:49.725779 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721451 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:14:49.725779 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721453 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:14:49.725779 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721457 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:14:49.725779 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721460 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:14:49.725779 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721463 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:14:49.725779 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721465 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:14:49.725779 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721468 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:14:49.725779 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721471 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:14:49.725779 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721474 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:14:49.725779 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721476 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:14:49.725779 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721480 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:14:49.726243 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721482 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:14:49.726243 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721485 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:14:49.726243 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721488 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:14:49.726243 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721491 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:14:49.726243 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721515 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:14:49.726243 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721519 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:14:49.726243 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721522 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:14:49.726243 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721524 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:14:49.726243 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721527 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:14:49.726243 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721530 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:14:49.726243 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721533 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:14:49.726243 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721536 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:14:49.726243 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721539 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:14:49.726243 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721542 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:14:49.726243 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721544 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:14:49.726243 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721547 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:14:49.726243 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721550 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:14:49.726243 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721552 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:14:49.726243 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.721555 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:14:49.726243 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721631 2553 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721674 2553 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721682 2553 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721687 2553 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721693 2553 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721696 2553 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721700 2553 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721704 2553 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721707 2553 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721710 2553 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721714 2553 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721717 2553 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721721 2553 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721724 2553 flags.go:64] FLAG: --cgroup-root="" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721727 2553 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721730 2553 flags.go:64] FLAG: --client-ca-file="" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721733 2553 flags.go:64] FLAG: --cloud-provider="external" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721736 2553 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721740 2553 flags.go:64] FLAG: --cluster-domain="" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721743 2553 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721746 2553 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721749 2553 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721753 2553 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721756 2553 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 14:14:49.726750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721759 2553 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721762 2553 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721765 2553 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721768 2553 flags.go:64] FLAG: --contention-profiling="false" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721771 2553 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721774 2553 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721777 2553 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721780 2553 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721784 2553 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721787 2553 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721793 2553 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721796 2553 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721800 2553 flags.go:64] FLAG: --enable-server="true" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721803 2553 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721809 2553 flags.go:64] FLAG: --event-burst="100" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721811 2553 flags.go:64] FLAG: --event-qps="50" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721814 2553 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721817 2553 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721820 2553 flags.go:64] FLAG: --eviction-hard="" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721824 2553 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721827 2553 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721830 2553 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721833 2553 flags.go:64] FLAG: --eviction-soft="" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721836 2553 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721838 2553 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 14:14:49.727366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721843 2553 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721846 2553 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721848 2553 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721851 2553 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721854 2553 flags.go:64] FLAG: --feature-gates="" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721858 2553 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721861 2553 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721864 2553 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721867 2553 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721870 2553 flags.go:64] FLAG: --healthz-port="10248" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721873 2553 flags.go:64] FLAG: --help="false" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721876 2553 flags.go:64] FLAG: --hostname-override="ip-10-0-135-17.ec2.internal" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721879 2553 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721882 2553 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721885 2553 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721888 2553 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721892 2553 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721896 2553 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721899 2553 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721903 2553 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721906 2553 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721909 2553 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721912 2553 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721915 2553 flags.go:64] FLAG: --kube-reserved="" Sep 11 14:14:49.728004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721918 2553 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721921 2553 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721924 2553 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721927 2553 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721930 2553 flags.go:64] FLAG: --lock-file="" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721933 2553 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721935 2553 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721938 2553 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721944 2553 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721947 2553 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721950 2553 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721953 2553 flags.go:64] FLAG: --logging-format="text" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721955 2553 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721959 2553 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721962 2553 flags.go:64] FLAG: --manifest-url="" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721964 2553 flags.go:64] FLAG: --manifest-url-header="" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721968 2553 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721971 2553 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721976 2553 flags.go:64] FLAG: --max-pods="110" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721979 2553 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721982 2553 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721985 2553 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721988 2553 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721991 2553 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721994 2553 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 14:14:49.728624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.721999 2553 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722006 2553 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722011 2553 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722014 2553 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722017 2553 flags.go:64] FLAG: --pod-cidr="" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722020 2553 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722025 2553 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722028 2553 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722031 2553 flags.go:64] FLAG: --pods-per-core="0" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722034 2553 flags.go:64] FLAG: --port="10250" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722038 2553 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722041 2553 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03bb2026d06247901" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722044 2553 flags.go:64] FLAG: --qos-reserved="" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722047 2553 flags.go:64] FLAG: --read-only-port="10255" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722050 2553 flags.go:64] FLAG: --register-node="true" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722052 2553 flags.go:64] FLAG: --register-with-taints="" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722056 2553 flags.go:64] FLAG: --registry-burst="10" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722060 2553 flags.go:64] FLAG: --registry-qps="5" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722063 2553 flags.go:64] FLAG: --reserved-cpus="" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722065 2553 flags.go:64] FLAG: --reserved-memory="" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722069 2553 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722072 2553 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722075 2553 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722077 2553 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722080 2553 flags.go:64] FLAG: --runonce="false" Sep 11 14:14:49.729272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722083 2553 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722086 2553 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722089 2553 flags.go:64] FLAG: --seccomp-default="false" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722092 2553 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722095 2553 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722099 2553 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722102 2553 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722105 2553 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722109 2553 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722113 2553 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722116 2553 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722119 2553 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722122 2553 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722125 2553 flags.go:64] FLAG: --system-cgroups="" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722128 2553 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722133 2553 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722136 2553 flags.go:64] FLAG: --tls-cert-file="" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722139 2553 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722142 2553 flags.go:64] FLAG: --tls-min-version="" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722145 2553 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722148 2553 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722150 2553 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722153 2553 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722156 2553 flags.go:64] FLAG: --v="2" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722160 2553 flags.go:64] FLAG: --version="false" Sep 11 14:14:49.729909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722164 2553 flags.go:64] FLAG: --vmodule="" Sep 11 14:14:49.730544 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722173 2553 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 14:14:49.730544 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.722176 2553 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 14:14:49.730544 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722298 2553 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:14:49.730544 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722302 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:14:49.730544 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722305 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:14:49.730544 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722308 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:14:49.730544 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722311 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:14:49.730544 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722314 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:14:49.730544 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722317 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:14:49.730544 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722320 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:14:49.730544 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722322 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:14:49.730544 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722325 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:14:49.730544 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722328 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:14:49.730544 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722331 2553 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:14:49.730544 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722335 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:14:49.730544 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722339 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:14:49.730544 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722342 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:14:49.730544 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722345 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:14:49.730544 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722347 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:14:49.730544 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722350 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:14:49.731085 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722353 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:14:49.731085 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722355 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:14:49.731085 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722358 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:14:49.731085 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722361 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:14:49.731085 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722363 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:14:49.731085 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722366 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:14:49.731085 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722369 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:14:49.731085 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722371 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:14:49.731085 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722374 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:14:49.731085 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722376 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:14:49.731085 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722379 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:14:49.731085 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722382 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:14:49.731085 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722385 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:14:49.731085 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722388 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:14:49.731085 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722391 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:14:49.731085 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722393 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:14:49.731085 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722396 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:14:49.731085 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722399 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:14:49.731085 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722401 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:14:49.731085 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722404 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:14:49.731650 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722407 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:14:49.731650 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722425 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:14:49.731650 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722428 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:14:49.731650 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722431 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:14:49.731650 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722434 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:14:49.731650 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722436 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:14:49.731650 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722440 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:14:49.731650 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722445 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:14:49.731650 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722447 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:14:49.731650 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722450 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:14:49.731650 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722460 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:14:49.731650 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722463 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:14:49.731650 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722466 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:14:49.731650 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722469 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:14:49.731650 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722471 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:14:49.731650 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722476 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:14:49.731650 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722479 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:14:49.731650 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722482 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:14:49.731650 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722485 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:14:49.732122 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722488 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:14:49.732122 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722491 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:14:49.732122 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722494 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:14:49.732122 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722498 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:14:49.732122 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722502 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:14:49.732122 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722505 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:14:49.732122 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722508 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:14:49.732122 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722511 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:14:49.732122 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722514 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:14:49.732122 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722517 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:14:49.732122 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722520 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:14:49.732122 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722523 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:14:49.732122 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722526 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:14:49.732122 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722528 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:14:49.732122 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722531 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:14:49.732122 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722533 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:14:49.732122 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722536 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:14:49.732122 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722539 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:14:49.732122 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722541 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:14:49.732122 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722546 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:14:49.732643 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722550 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:14:49.732643 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722553 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:14:49.732643 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722555 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:14:49.732643 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722558 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:14:49.732643 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722561 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:14:49.732643 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722564 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:14:49.732643 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722566 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:14:49.732643 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722569 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:14:49.732643 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722573 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:14:49.732643 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722576 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:14:49.732643 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722579 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:14:49.732643 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722582 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:14:49.732643 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722585 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:14:49.732643 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722588 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:14:49.732643 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722591 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:14:49.732643 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722593 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:14:49.732643 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722596 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:14:49.732643 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722598 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:14:49.732643 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722601 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:14:49.733115 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722604 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:14:49.733115 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722607 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:14:49.733115 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.722610 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:14:49.733115 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.723152 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 14:14:49.733115 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.730582 2553 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 14:14:49.733115 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.730597 2553 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 14:14:49.733115 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.731441 2553 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 14:14:49.733115 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.731452 2553 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 14:14:49.733115 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731507 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:14:49.733115 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731512 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:14:49.733115 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731515 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:14:49.733115 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731518 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:14:49.733115 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731521 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:14:49.733115 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731524 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:14:49.733115 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731526 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:14:49.733498 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731529 2553 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:14:49.733498 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731532 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:14:49.733498 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731535 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:14:49.733498 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731538 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:14:49.733498 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731540 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:14:49.733498 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731543 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:14:49.733498 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731545 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:14:49.733498 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731548 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:14:49.733498 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731551 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:14:49.733498 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731553 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:14:49.733498 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731556 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:14:49.733498 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731559 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:14:49.733498 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731562 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:14:49.733498 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731565 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:14:49.733498 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731567 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:14:49.733498 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731570 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:14:49.733498 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731573 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:14:49.733498 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731575 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:14:49.733498 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731578 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:14:49.733947 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731581 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:14:49.733947 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731585 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:14:49.733947 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731589 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:14:49.733947 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731593 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:14:49.733947 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731596 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:14:49.733947 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731598 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:14:49.733947 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731601 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:14:49.733947 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731603 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:14:49.733947 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731606 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:14:49.733947 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731610 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:14:49.733947 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731613 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:14:49.733947 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731616 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:14:49.733947 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731618 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:14:49.733947 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731621 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:14:49.733947 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731624 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:14:49.733947 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731627 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:14:49.733947 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731629 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:14:49.733947 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731631 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:14:49.733947 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731634 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:14:49.734587 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731637 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:14:49.734587 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731639 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:14:49.734587 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731642 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:14:49.734587 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731644 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:14:49.734587 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731647 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:14:49.734587 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731650 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:14:49.734587 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731652 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:14:49.734587 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731655 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:14:49.734587 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731658 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:14:49.734587 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731660 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:14:49.734587 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731663 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:14:49.734587 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731666 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:14:49.734587 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731669 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:14:49.734587 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731671 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:14:49.734587 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731674 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:14:49.734587 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731677 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:14:49.734587 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731681 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:14:49.734587 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731684 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:14:49.734587 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731687 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:14:49.734587 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731690 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:14:49.735075 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731692 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:14:49.735075 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731695 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:14:49.735075 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731698 2553 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:14:49.735075 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731700 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:14:49.735075 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731703 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:14:49.735075 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731705 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:14:49.735075 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731708 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:14:49.735075 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731710 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:14:49.735075 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731714 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:14:49.735075 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731718 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:14:49.735075 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731721 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:14:49.735075 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731724 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:14:49.735075 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731727 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:14:49.735075 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731729 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:14:49.735075 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731732 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:14:49.735075 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731736 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:14:49.735075 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731739 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:14:49.735075 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731742 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:14:49.735075 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731744 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:14:49.735075 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731747 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:14:49.735604 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731750 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:14:49.735604 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731753 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:14:49.735604 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731756 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:14:49.735604 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731758 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:14:49.735604 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731761 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:14:49.735604 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731764 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:14:49.735604 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731766 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:14:49.735604 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731769 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:14:49.735604 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731772 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:14:49.735604 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731775 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:14:49.735604 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731778 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:14:49.735604 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731781 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:14:49.735604 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731783 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:14:49.735604 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731785 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:14:49.735604 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.731790 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 14:14:49.735974 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731926 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 14:14:49.735974 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731932 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 14:14:49.735974 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731935 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 14:14:49.735974 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731937 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 14:14:49.735974 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731941 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 14:14:49.735974 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731943 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 14:14:49.735974 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731946 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 14:14:49.735974 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731949 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 14:14:49.735974 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731951 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 14:14:49.735974 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731954 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 14:14:49.735974 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731956 2553 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 14:14:49.735974 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731960 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 14:14:49.735974 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731964 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 14:14:49.735974 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731967 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 14:14:49.735974 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731970 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 14:14:49.735974 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731972 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 14:14:49.735974 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731975 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 14:14:49.735974 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731978 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 14:14:49.735974 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731981 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 14:14:49.736462 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731984 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 14:14:49.736462 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731986 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 14:14:49.736462 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731989 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 14:14:49.736462 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731992 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 14:14:49.736462 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731995 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 14:14:49.736462 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.731997 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 14:14:49.736462 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732001 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 14:14:49.736462 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732004 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 14:14:49.736462 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732006 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 14:14:49.736462 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732010 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 14:14:49.736462 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732012 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 14:14:49.736462 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732015 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 14:14:49.736462 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732017 2553 feature_gate.go:328] unrecognized feature gate: Example Sep 11 14:14:49.736462 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732020 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 14:14:49.736462 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732022 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 14:14:49.736462 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732025 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 14:14:49.736462 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732027 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 14:14:49.736462 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732030 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 14:14:49.736462 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732032 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 14:14:49.736462 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732035 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 14:14:49.736949 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732038 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 14:14:49.736949 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732040 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 14:14:49.736949 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732042 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 14:14:49.736949 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732045 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 14:14:49.736949 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732047 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 14:14:49.736949 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732050 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 14:14:49.736949 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732052 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 14:14:49.736949 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732055 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 14:14:49.736949 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732057 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 14:14:49.736949 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732059 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 14:14:49.736949 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732062 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 14:14:49.736949 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732065 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 14:14:49.736949 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732067 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 14:14:49.736949 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732069 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 14:14:49.736949 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732072 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 14:14:49.736949 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732074 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 14:14:49.736949 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732077 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 14:14:49.736949 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732079 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 14:14:49.736949 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732082 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 14:14:49.736949 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732085 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 14:14:49.737448 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732087 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 14:14:49.737448 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732090 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 14:14:49.737448 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732093 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 14:14:49.737448 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732095 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 14:14:49.737448 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732098 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 14:14:49.737448 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732100 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 14:14:49.737448 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732102 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 14:14:49.737448 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732105 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 14:14:49.737448 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732107 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 14:14:49.737448 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732110 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 14:14:49.737448 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732112 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 14:14:49.737448 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732115 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 14:14:49.737448 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732117 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 14:14:49.737448 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732120 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 14:14:49.737448 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732123 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 14:14:49.737448 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732126 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 14:14:49.737448 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732128 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 14:14:49.737448 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732131 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 14:14:49.737448 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732134 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 14:14:49.737448 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732136 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 14:14:49.737932 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732139 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 14:14:49.737932 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732141 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 14:14:49.737932 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732144 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 14:14:49.737932 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732146 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 14:14:49.737932 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732148 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 14:14:49.737932 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732151 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 14:14:49.737932 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732153 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 14:14:49.737932 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732156 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 14:14:49.737932 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732158 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 14:14:49.737932 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732161 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 14:14:49.737932 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732164 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 14:14:49.737932 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732166 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 14:14:49.737932 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732169 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 14:14:49.737932 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732173 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 14:14:49.737932 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732176 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 14:14:49.737932 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732180 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 14:14:49.737932 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732183 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 14:14:49.737932 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732186 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 14:14:49.737932 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732188 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 14:14:49.737932 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:49.732192 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 14:14:49.738461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.732197 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 14:14:49.738461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.732307 2553 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 14:14:49.738461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.734218 2553 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 14:14:49.738461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.735692 2553 server.go:1019] "Starting client certificate rotation" Sep 11 14:14:49.738461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.735794 2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 14:14:49.738461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.735823 2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 14:14:49.738461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.738111 2553 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 14:14:49.739615 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.739598 2553 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 14:14:49.749950 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.749928 2553 log.go:25] "Validated CRI v1 runtime API" Sep 11 14:14:49.753779 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.753763 2553 log.go:25] "Validated CRI v1 image API" Sep 11 14:14:49.755872 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.755857 2553 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 14:14:49.760833 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.760816 2553 fs.go:135] Filesystem UUIDs: map[745a174b-7d02-4c95-a429-9b5f555fb44d:/dev/nvme0n1p3 789adde0-2e13-4da4-b2a4-34aef97a74e9:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Sep 11 14:14:49.760876 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.760833 2553 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 14:14:49.767316 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.767190 2553 manager.go:216] Machine: {Timestamp:2026-09-11 14:14:49.765610164 +0000 UTC m=+0.379716500 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3114280 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23a0540fc3434b7147737dfe26137f SystemUUID:ec23a054-0fc3-434b-7147-737dfe26137f BootID:bbde3141-7898-4999-afc3-cfeff7ad1e1b Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:70:0d:be:69:39 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:70:0d:be:69:39 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:ef:86:ff:28:67 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 14:14:49.767316 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.767303 2553 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 14:14:49.767466 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.767388 2553 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 14:14:49.767993 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.767978 2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 14:14:49.770940 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.770911 2553 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 14:14:49.771113 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.770945 2553 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-17.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 14:14:49.771743 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.771730 2553 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 14:14:49.771795 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.771749 2553 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 14:14:49.771795 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.771760 2553 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 14:14:49.772610 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.772599 2553 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 14:14:49.773495 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.773484 2553 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 14:14:49.777174 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.777161 2553 state_mem.go:36] "Initialized new in-memory state store" Sep 11 14:14:49.777302 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.777291 2553 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 14:14:49.777358 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.777332 2553 kubelet.go:486] "Attempting to sync node with API server" Sep 11 14:14:49.777358 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.777353 2553 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 14:14:49.777456 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.777369 2553 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 14:14:49.777456 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.777383 2553 kubelet.go:398] "Adding apiserver pod source" Sep 11 14:14:49.777456 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.777397 2553 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 14:14:49.777781 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.777767 2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 14:14:49.777848 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.777792 2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 14:14:49.780730 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.780715 2553 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 14:14:49.782989 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.782976 2553 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 14:14:49.783040 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.783006 2553 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 14:14:49.784697 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.784685 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 14:14:49.784737 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.784701 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 14:14:49.784737 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.784708 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 14:14:49.784737 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.784714 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 14:14:49.784737 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.784724 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 14:14:49.784737 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.784734 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 14:14:49.784864 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.784752 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 14:14:49.784864 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.784757 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 14:14:49.784864 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.784764 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 14:14:49.784864 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.784769 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 14:14:49.784864 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.784778 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 14:14:49.784864 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.784783 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 14:14:49.785599 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.785590 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 14:14:49.785599 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.785598 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 14:14:49.789339 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.789325 2553 server.go:1268] "Started kubelet" Sep 11 14:14:49.789406 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.789374 2553 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 14:14:49.789481 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.789427 2553 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 14:14:49.789538 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.789506 2553 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 14:14:49.790299 ip-10-0-135-17 systemd[1]: Started Kubernetes Kubelet. Sep 11 14:14:49.790622 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.790608 2553 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 14:14:49.790689 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:49.790636 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 14:14:49.790773 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.790760 2553 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-17.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 14:14:49.791810 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:49.791314 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-17.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 14:14:49.792299 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.792279 2553 server.go:310] "Adding debug handlers to kubelet server" Sep 11 14:14:49.799268 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:49.799235 2553 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 14:14:49.801346 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:49.800285 2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-17.ec2.internal.18d449cb5eb8eefe default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-17.ec2.internal,UID:ip-10-0-135-17.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-17.ec2.internal,},FirstTimestamp:2026-09-11 14:14:49.789296382 +0000 UTC m=+0.403402718,LastTimestamp:2026-09-11 14:14:49.789296382 +0000 UTC m=+0.403402718,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-17.ec2.internal,}" Sep 11 14:14:49.802074 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.802058 2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 14:14:49.802728 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.802702 2553 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 14:14:49.802873 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:49.802852 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-17.ec2.internal\" not found" Sep 11 14:14:49.803397 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.803371 2553 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 14:14:49.803397 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.803374 2553 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 14:14:49.803537 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.803407 2553 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 14:14:49.803537 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.803519 2553 reconstruct.go:97] "Volume reconstruction finished" Sep 11 14:14:49.803537 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.803525 2553 reconciler.go:29] "Reconciler: start to sync state" Sep 11 14:14:49.804112 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.804089 2553 factory.go:55] Registering systemd factory Sep 11 14:14:49.804112 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.804105 2553 factory.go:223] Registration of the systemd container factory successfully Sep 11 14:14:49.804325 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.804312 2553 factory.go:153] Registering CRI-O factory Sep 11 14:14:49.804325 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.804327 2553 factory.go:223] Registration of the crio container factory successfully Sep 11 14:14:49.804446 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.804401 2553 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 14:14:49.804446 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.804432 2553 factory.go:103] Registering Raw factory Sep 11 14:14:49.804446 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.804447 2553 manager.go:1196] Started watching for new ooms in manager Sep 11 14:14:49.804847 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.804834 2553 manager.go:349] Starting recovery of all containers Sep 11 14:14:49.807089 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:49.807056 2553 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-17.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 11 14:14:49.807212 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:49.807195 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 11 14:14:49.817017 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.816855 2553 manager.go:354] Recovery completed Sep 11 14:14:49.821146 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.821126 2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gnqgd" Sep 11 14:14:49.821702 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.821691 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:14:49.823646 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.823631 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-17.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:14:49.823706 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.823661 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-17.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:14:49.823706 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.823675 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-17.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:14:49.824281 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.824266 2553 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 14:14:49.824281 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.824279 2553 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 14:14:49.824355 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.824294 2553 state_mem.go:36] "Initialized new in-memory state store" Sep 11 14:14:49.824355 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.824272 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:14:49.825100 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.825083 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-17.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:14:49.825187 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.825120 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-17.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:14:49.825187 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.825136 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-17.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:14:49.826095 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:49.825962 2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-17.ec2.internal.18d449cb60c514ac default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-17.ec2.internal,UID:ip-10-0-135-17.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-17.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-17.ec2.internal,},FirstTimestamp:2026-09-11 14:14:49.823646892 +0000 UTC m=+0.437753234,LastTimestamp:2026-09-11 14:14:49.823646892 +0000 UTC m=+0.437753234,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-17.ec2.internal,}" Sep 11 14:14:49.826550 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.826531 2553 policy_none.go:49] "None policy: Start" Sep 11 14:14:49.826633 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.826554 2553 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 14:14:49.826633 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.826567 2553 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 14:14:49.828585 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.828572 2553 policy_none.go:47] "Start" Sep 11 14:14:49.829141 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.829125 2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gnqgd" Sep 11 14:14:49.864160 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.864123 2553 manager.go:337] "Starting Device Plugin manager" Sep 11 14:14:49.884097 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:49.864192 2553 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 14:14:49.884097 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.864223 2553 server.go:87] "Starting device plugin registration server" Sep 11 14:14:49.884097 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.864691 2553 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 14:14:49.884097 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.864707 2553 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 14:14:49.884097 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.864837 2553 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 14:14:49.884097 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.864911 2553 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 14:14:49.884097 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.864919 2553 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 14:14:49.884097 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:49.865579 2553 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 14:14:49.884097 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:49.865628 2553 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-17.ec2.internal\" not found" Sep 11 14:14:49.904117 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.904088 2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 14:14:49.905326 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.905312 2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 14:14:49.905389 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.905366 2553 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 14:14:49.905450 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.905390 2553 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 14:14:49.905513 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:49.905497 2553 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 14:14:49.909670 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.909653 2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:14:49.965944 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.965840 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:14:49.966890 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.966872 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-17.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:14:49.966995 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.966904 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-17.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:14:49.966995 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.966914 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-17.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:14:49.966995 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.966940 2553 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-17.ec2.internal" Sep 11 14:14:49.975479 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:49.975456 2553 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-17.ec2.internal" Sep 11 14:14:49.975479 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:49.975479 2553 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-17.ec2.internal\": node \"ip-10-0-135-17.ec2.internal\" not found" Sep 11 14:14:50.005691 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.005663 2553 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-17.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-17.ec2.internal"] Sep 11 14:14:50.005819 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.005729 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:14:50.007235 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.007214 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-17.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:14:50.007349 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.007242 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-17.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:14:50.007349 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.007253 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-17.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:14:50.007601 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:50.007582 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-17.ec2.internal\" not found" Sep 11 14:14:50.009670 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.009658 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:14:50.009791 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.009776 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-17.ec2.internal" Sep 11 14:14:50.009853 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.009805 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:14:50.010375 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.010356 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-17.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:14:50.010496 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.010359 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-17.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:14:50.010496 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.010406 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-17.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:14:50.010496 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.010441 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-17.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:14:50.010496 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.010386 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-17.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:14:50.010496 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.010473 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-17.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:14:50.012631 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.012615 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-17.ec2.internal" Sep 11 14:14:50.012724 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.012640 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 14:14:50.013348 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.013333 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-17.ec2.internal" event="NodeHasSufficientMemory" Sep 11 14:14:50.013452 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.013362 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-17.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 14:14:50.013452 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.013376 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-17.ec2.internal" event="NodeHasSufficientPID" Sep 11 14:14:50.040079 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:50.040052 2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-17.ec2.internal\" not found" node="ip-10-0-135-17.ec2.internal" Sep 11 14:14:50.044512 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:50.044496 2553 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-17.ec2.internal\" not found" node="ip-10-0-135-17.ec2.internal" Sep 11 14:14:50.105248 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.105222 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eca2a8aca32a15dbbbb325138183d766-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-17.ec2.internal\" (UID: \"eca2a8aca32a15dbbbb325138183d766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-17.ec2.internal" Sep 11 14:14:50.105376 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.105254 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eca2a8aca32a15dbbbb325138183d766-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-17.ec2.internal\" (UID: \"eca2a8aca32a15dbbbb325138183d766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-17.ec2.internal" Sep 11 14:14:50.105376 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.105273 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7c73b8b4f6cacbe91fc9321bf2769db6-config\") pod \"kube-apiserver-proxy-ip-10-0-135-17.ec2.internal\" (UID: \"7c73b8b4f6cacbe91fc9321bf2769db6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-17.ec2.internal" Sep 11 14:14:50.108645 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:50.108629 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-17.ec2.internal\" not found" Sep 11 14:14:50.206022 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.205991 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eca2a8aca32a15dbbbb325138183d766-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-17.ec2.internal\" (UID: \"eca2a8aca32a15dbbbb325138183d766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-17.ec2.internal" Sep 11 14:14:50.206162 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.206024 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eca2a8aca32a15dbbbb325138183d766-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-17.ec2.internal\" (UID: \"eca2a8aca32a15dbbbb325138183d766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-17.ec2.internal" Sep 11 14:14:50.206162 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.206047 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7c73b8b4f6cacbe91fc9321bf2769db6-config\") pod \"kube-apiserver-proxy-ip-10-0-135-17.ec2.internal\" (UID: \"7c73b8b4f6cacbe91fc9321bf2769db6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-17.ec2.internal" Sep 11 14:14:50.206162 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.206112 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/eca2a8aca32a15dbbbb325138183d766-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-17.ec2.internal\" (UID: \"eca2a8aca32a15dbbbb325138183d766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-17.ec2.internal" Sep 11 14:14:50.206162 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.206112 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eca2a8aca32a15dbbbb325138183d766-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-17.ec2.internal\" (UID: \"eca2a8aca32a15dbbbb325138183d766\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-17.ec2.internal" Sep 11 14:14:50.206162 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.206120 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7c73b8b4f6cacbe91fc9321bf2769db6-config\") pod \"kube-apiserver-proxy-ip-10-0-135-17.ec2.internal\" (UID: \"7c73b8b4f6cacbe91fc9321bf2769db6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-17.ec2.internal" Sep 11 14:14:50.209089 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:50.209074 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-17.ec2.internal\" not found" Sep 11 14:14:50.309906 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:50.309833 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-17.ec2.internal\" not found" Sep 11 14:14:50.343661 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.343635 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-17.ec2.internal" Sep 11 14:14:50.347531 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.347510 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-17.ec2.internal" Sep 11 14:14:50.410168 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:50.410131 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-17.ec2.internal\" not found" Sep 11 14:14:50.511149 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:50.511111 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-17.ec2.internal\" not found" Sep 11 14:14:50.611271 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:50.611201 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-17.ec2.internal\" not found" Sep 11 14:14:50.712309 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:50.712276 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-17.ec2.internal\" not found" Sep 11 14:14:50.725248 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.725226 2553 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:14:50.734755 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.734737 2553 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 14:14:50.734862 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.734845 2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 14:14:50.734916 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.734880 2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 14:14:50.778368 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.778344 2553 apiserver.go:52] "Watching apiserver" Sep 11 14:14:50.789670 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.789650 2553 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:14:50.792326 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.792306 2553 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 14:14:50.794812 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.794762 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-bn8wp","openshift-multus/network-metrics-daemon-s2sfs","openshift-network-diagnostics/network-check-target-jgdkq","openshift-network-operator/iptables-alerter-ldh4b","kube-system/global-pull-secret-syncer-n7x6t","kube-system/konnectivity-agent-4zsn9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8","openshift-cluster-node-tuning-operator/tuned-t89f2","openshift-multus/multus-additional-cni-plugins-8jm26","openshift-multus/multus-t5h5f","openshift-ovn-kubernetes/ovnkube-node-mmqjh","openshift-dns/node-resolver-q25fm"] Sep 11 14:14:50.798653 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.798631 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bn8wp" Sep 11 14:14:50.801347 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.801325 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 14:14:50.801455 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.801437 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 14:14:50.801541 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.801471 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:14:50.801606 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.801543 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kfss7\"" Sep 11 14:14:50.801606 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.801544 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 14:14:50.801606 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:50.801574 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2sfs" podUID="05960fb2-54e8-48b2-852b-caabc3c413ca" Sep 11 14:14:50.802235 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.802219 2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 14:14:50.803374 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.803357 2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-17.ec2.internal" Sep 11 14:14:50.804338 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.804321 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:14:50.804425 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:50.804383 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgdkq" podUID="7a80c038-1fe1-4564-875f-e64fa2c5b2a6" Sep 11 14:14:50.807186 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.807168 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ldh4b" Sep 11 14:14:50.810128 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.810107 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:14:50.810210 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:50.810186 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7x6t" podUID="bbb4cb01-8e27-4823-9410-d5b11334c3a9" Sep 11 14:14:50.810891 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.810868 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lnk52\"" Sep 11 14:14:50.811101 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.811088 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 14:14:50.811183 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.811167 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs\") pod \"network-metrics-daemon-s2sfs\" (UID: \"05960fb2-54e8-48b2-852b-caabc3c413ca\") " pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:14:50.811220 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.811198 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z94lx\" (UniqueName: \"kubernetes.io/projected/05960fb2-54e8-48b2-852b-caabc3c413ca-kube-api-access-z94lx\") pod \"network-metrics-daemon-s2sfs\" (UID: \"05960fb2-54e8-48b2-852b-caabc3c413ca\") " pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:14:50.811254 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.811226 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5159b394-744e-49c1-932d-a78367218c21-host\") pod \"node-ca-bn8wp\" (UID: \"5159b394-744e-49c1-932d-a78367218c21\") " pod="openshift-image-registry/node-ca-bn8wp" Sep 11 14:14:50.811254 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.811241 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5159b394-744e-49c1-932d-a78367218c21-serviceca\") pod \"node-ca-bn8wp\" (UID: \"5159b394-744e-49c1-932d-a78367218c21\") " pod="openshift-image-registry/node-ca-bn8wp" Sep 11 14:14:50.811319 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.811259 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqz4f\" (UniqueName: \"kubernetes.io/projected/5159b394-744e-49c1-932d-a78367218c21-kube-api-access-tqz4f\") pod \"node-ca-bn8wp\" (UID: \"5159b394-744e-49c1-932d-a78367218c21\") " pod="openshift-image-registry/node-ca-bn8wp" Sep 11 14:14:50.811319 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.811284 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdd8n\" (UniqueName: \"kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n\") pod \"network-check-target-jgdkq\" (UID: \"7a80c038-1fe1-4564-875f-e64fa2c5b2a6\") " pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:14:50.811873 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.811858 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:14:50.811959 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.811936 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 14:14:50.812849 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.812836 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4zsn9" Sep 11 14:14:50.815717 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.815697 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 14:14:50.815838 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.815757 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 14:14:50.815838 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.815827 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:50.815947 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.815895 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-j6l7d\"" Sep 11 14:14:50.818009 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.817993 2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 14:14:50.818750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.818736 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:50.819098 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.819081 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 14:14:50.819341 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.819326 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vhfbv\"" Sep 11 14:14:50.819616 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.819601 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 14:14:50.819746 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.819732 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 14:14:50.821598 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.821581 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:50.822334 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.822315 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:14:50.822545 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.822470 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sxttc\"" Sep 11 14:14:50.822679 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.822667 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 14:14:50.824574 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.824557 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t5h5f" Sep 11 14:14:50.824737 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.824722 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 14:14:50.824909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.824890 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 14:14:50.825418 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.825401 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vwmwk\"" Sep 11 14:14:50.825521 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.825427 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 14:14:50.825521 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.825433 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 14:14:50.825521 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.825403 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 14:14:50.827523 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.827505 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:50.828065 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.828049 2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 14:14:50.828153 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.828129 2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-17.ec2.internal" Sep 11 14:14:50.830165 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.830143 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q25fm" Sep 11 14:14:50.830962 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.830939 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 14:14:50.831056 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.831013 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-k455g\"" Sep 11 14:14:50.831133 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.831092 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 14:14:50.831257 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.831240 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 14:14:50.831411 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.831396 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 14:14:50.831582 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.831563 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rcqb6\"" Sep 11 14:14:50.831738 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.831721 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-17.ec2.internal"] Sep 11 14:14:50.832125 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.832109 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 14:14:50.832287 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.832273 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 14:14:50.832541 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.832525 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 14:14:50.833002 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.832985 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 14:14:50.833218 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.833205 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fnjk8\"" Sep 11 14:14:50.833389 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.833346 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 14:09:50 +0000 UTC" deadline="2028-05-02 16:24:38.705184836 +0000 UTC" Sep 11 14:14:50.833440 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.833389 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14378h9m47.871798335s" Sep 11 14:14:50.833954 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.833939 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 14:14:50.838160 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.838142 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-17.ec2.internal"] Sep 11 14:14:50.838576 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.838564 2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 14:14:50.854790 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.854769 2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fb5l4" Sep 11 14:14:50.866336 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.866290 2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fb5l4" Sep 11 14:14:50.904404 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.904381 2553 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 14:14:50.911707 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.911684 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z94lx\" (UniqueName: \"kubernetes.io/projected/05960fb2-54e8-48b2-852b-caabc3c413ca-kube-api-access-z94lx\") pod \"network-metrics-daemon-s2sfs\" (UID: \"05960fb2-54e8-48b2-852b-caabc3c413ca\") " pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:14:50.911817 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.911716 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbb4cb01-8e27-4823-9410-d5b11334c3a9-kubelet-config\") pod \"global-pull-secret-syncer-n7x6t\" (UID: \"bbb4cb01-8e27-4823-9410-d5b11334c3a9\") " pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:14:50.911817 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.911733 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-host-kubelet\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:50.911817 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.911750 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f3ef9965-9ec2-465d-b450-84ba57947908-iptables-alerter-script\") pod \"iptables-alerter-ldh4b\" (UID: \"f3ef9965-9ec2-465d-b450-84ba57947908\") " pod="openshift-network-operator/iptables-alerter-ldh4b" Sep 11 14:14:50.911817 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.911768 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqpkt\" (UniqueName: \"kubernetes.io/projected/f3ef9965-9ec2-465d-b450-84ba57947908-kube-api-access-nqpkt\") pod \"iptables-alerter-ldh4b\" (UID: \"f3ef9965-9ec2-465d-b450-84ba57947908\") " pod="openshift-network-operator/iptables-alerter-ldh4b" Sep 11 14:14:50.912034 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.911830 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c35b9597-006b-4b14-97e5-9e7d06ed8cd4-device-dir\") pod \"aws-ebs-csi-driver-node-9cgn8\" (UID: \"c35b9597-006b-4b14-97e5-9e7d06ed8cd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:50.912034 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.911922 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7a24be49-54d5-46bf-b7ed-73b43b0bcb54-hosts-file\") pod \"node-resolver-q25fm\" (UID: \"7a24be49-54d5-46bf-b7ed-73b43b0bcb54\") " pod="openshift-dns/node-resolver-q25fm" Sep 11 14:14:50.912034 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.911963 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a24be49-54d5-46bf-b7ed-73b43b0bcb54-tmp-dir\") pod \"node-resolver-q25fm\" (UID: \"7a24be49-54d5-46bf-b7ed-73b43b0bcb54\") " pod="openshift-dns/node-resolver-q25fm" Sep 11 14:14:50.912034 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.911999 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-etc-sysconfig\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:50.912034 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912024 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-host\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:50.912220 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912046 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkr7m\" (UniqueName: \"kubernetes.io/projected/1cc0d3d9-09b0-48f9-99b7-2a11488eed7a-kube-api-access-gkr7m\") pod \"multus-additional-cni-plugins-8jm26\" (UID: \"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a\") " pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:50.912220 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912064 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c35b9597-006b-4b14-97e5-9e7d06ed8cd4-registration-dir\") pod \"aws-ebs-csi-driver-node-9cgn8\" (UID: \"c35b9597-006b-4b14-97e5-9e7d06ed8cd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:50.912220 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912086 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs\") pod \"network-metrics-daemon-s2sfs\" (UID: \"05960fb2-54e8-48b2-852b-caabc3c413ca\") " pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:14:50.912220 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912101 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-sys\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:50.912220 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912124 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b1166baf-c7df-4e63-9360-81dec348a479-konnectivity-ca\") pod \"konnectivity-agent-4zsn9\" (UID: \"b1166baf-c7df-4e63-9360-81dec348a479\") " pod="kube-system/konnectivity-agent-4zsn9" Sep 11 14:14:50.912220 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912177 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c35b9597-006b-4b14-97e5-9e7d06ed8cd4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9cgn8\" (UID: \"c35b9597-006b-4b14-97e5-9e7d06ed8cd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:50.912220 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912201 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6288df4f-7888-432f-b13f-463cb6d8cdd0-etc-tuned\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:50.912220 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:50.912213 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:14:50.912570 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912235 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-host-cni-bin\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:50.912570 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:50.912322 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs podName:05960fb2-54e8-48b2-852b-caabc3c413ca nodeName:}" failed. No retries permitted until 2026-09-11 14:14:51.412267948 +0000 UTC m=+2.026374277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs") pod "network-metrics-daemon-s2sfs" (UID: "05960fb2-54e8-48b2-852b-caabc3c413ca") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:14:50.912570 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912337 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-host-cni-netd\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:50.912570 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912353 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b2452d9-1730-458c-b380-dbcfd9e5be19-env-overrides\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:50.912570 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912371 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-etc-sysctl-d\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:50.912570 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912389 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-systemd-units\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:50.912570 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912412 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-etc-modprobe-d\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:50.912570 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912429 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm8kj\" (UniqueName: \"kubernetes.io/projected/6288df4f-7888-432f-b13f-463cb6d8cdd0-kube-api-access-xm8kj\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:50.912570 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912467 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-system-cni-dir\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:50.912570 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912502 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-host-var-lib-cni-multus\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:50.912570 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912543 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-host-var-lib-kubelet\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:50.912570 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912566 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-run-systemd\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:50.913019 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912582 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1cc0d3d9-09b0-48f9-99b7-2a11488eed7a-system-cni-dir\") pod \"multus-additional-cni-plugins-8jm26\" (UID: \"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a\") " pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:50.913019 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912598 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-node-log\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:50.913019 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912615 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:50.913019 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912632 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b2452d9-1730-458c-b380-dbcfd9e5be19-ovn-node-metrics-cert\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:50.913019 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912648 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3ef9965-9ec2-465d-b450-84ba57947908-host-slash\") pod \"iptables-alerter-ldh4b\" (UID: \"f3ef9965-9ec2-465d-b450-84ba57947908\") " pod="openshift-network-operator/iptables-alerter-ldh4b" Sep 11 14:14:50.913019 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912669 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c35b9597-006b-4b14-97e5-9e7d06ed8cd4-socket-dir\") pod \"aws-ebs-csi-driver-node-9cgn8\" (UID: \"c35b9597-006b-4b14-97e5-9e7d06ed8cd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:50.913019 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912695 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-etc-sysctl-conf\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:50.913019 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912716 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-run\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:50.913019 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912733 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1cc0d3d9-09b0-48f9-99b7-2a11488eed7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8jm26\" (UID: \"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a\") " pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:50.913019 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912758 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-run-ovn\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:50.913019 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912778 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-etc-systemd\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:50.913019 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912795 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-var-lib-kubelet\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:50.913019 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912808 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed3f15f3-e064-4e2e-972b-07e824971be7-cni-binary-copy\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:50.913019 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912823 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-host-run-netns\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:50.913019 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912839 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlcfx\" (UniqueName: \"kubernetes.io/projected/ed3f15f3-e064-4e2e-972b-07e824971be7-kube-api-access-dlcfx\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:50.913019 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912866 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-run-openvswitch\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:50.913720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912884 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b2452d9-1730-458c-b380-dbcfd9e5be19-ovnkube-config\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:50.913720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912900 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b2452d9-1730-458c-b380-dbcfd9e5be19-ovnkube-script-lib\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:50.913720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912918 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-host-run-multus-certs\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:50.913720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912932 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6288df4f-7888-432f-b13f-463cb6d8cdd0-tmp\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:50.913720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912948 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-host-run-k8s-cni-cncf-io\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:50.913720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912964 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-host-var-lib-cni-bin\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:50.913720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.912986 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-multus-conf-dir\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:50.913720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913013 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9df8w\" (UniqueName: \"kubernetes.io/projected/c35b9597-006b-4b14-97e5-9e7d06ed8cd4-kube-api-access-9df8w\") pod \"aws-ebs-csi-driver-node-9cgn8\" (UID: \"c35b9597-006b-4b14-97e5-9e7d06ed8cd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:50.913720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913031 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbb4cb01-8e27-4823-9410-d5b11334c3a9-dbus\") pod \"global-pull-secret-syncer-n7x6t\" (UID: \"bbb4cb01-8e27-4823-9410-d5b11334c3a9\") " pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:14:50.913720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913047 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-multus-socket-dir-parent\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:50.913720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913068 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-etc-openvswitch\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:50.913720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913087 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-log-socket\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:50.913720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913103 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c35b9597-006b-4b14-97e5-9e7d06ed8cd4-etc-selinux\") pod \"aws-ebs-csi-driver-node-9cgn8\" (UID: \"c35b9597-006b-4b14-97e5-9e7d06ed8cd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:50.913720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913141 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-lib-modules\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:50.913720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913164 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-multus-cni-dir\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:50.913720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913182 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed3f15f3-e064-4e2e-972b-07e824971be7-multus-daemon-config\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:50.914305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913210 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret\") pod \"global-pull-secret-syncer-n7x6t\" (UID: \"bbb4cb01-8e27-4823-9410-d5b11334c3a9\") " pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:14:50.914305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913227 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-etc-kubernetes\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:50.914305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913242 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-host-slash\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:50.914305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913258 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1cc0d3d9-09b0-48f9-99b7-2a11488eed7a-cnibin\") pod \"multus-additional-cni-plugins-8jm26\" (UID: \"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a\") " pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:50.914305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913276 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5159b394-744e-49c1-932d-a78367218c21-host\") pod \"node-ca-bn8wp\" (UID: \"5159b394-744e-49c1-932d-a78367218c21\") " pod="openshift-image-registry/node-ca-bn8wp" Sep 11 14:14:50.914305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913315 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5159b394-744e-49c1-932d-a78367218c21-host\") pod \"node-ca-bn8wp\" (UID: \"5159b394-744e-49c1-932d-a78367218c21\") " pod="openshift-image-registry/node-ca-bn8wp" Sep 11 14:14:50.914305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913312 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqz4f\" (UniqueName: \"kubernetes.io/projected/5159b394-744e-49c1-932d-a78367218c21-kube-api-access-tqz4f\") pod \"node-ca-bn8wp\" (UID: \"5159b394-744e-49c1-932d-a78367218c21\") " pod="openshift-image-registry/node-ca-bn8wp" Sep 11 14:14:50.914305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913346 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-host-run-netns\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:50.914305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913375 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-host-run-ovn-kubernetes\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:50.914305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913485 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b1166baf-c7df-4e63-9360-81dec348a479-agent-certs\") pod \"konnectivity-agent-4zsn9\" (UID: \"b1166baf-c7df-4e63-9360-81dec348a479\") " pod="kube-system/konnectivity-agent-4zsn9" Sep 11 14:14:50.914305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913520 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1cc0d3d9-09b0-48f9-99b7-2a11488eed7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-8jm26\" (UID: \"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a\") " pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:50.914305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913546 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1cc0d3d9-09b0-48f9-99b7-2a11488eed7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8jm26\" (UID: \"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a\") " pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:50.914305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913571 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-cnibin\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:50.914305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913595 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-hostroot\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:50.914305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913618 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-etc-kubernetes\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:50.914305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913642 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxjjz\" (UniqueName: \"kubernetes.io/projected/6b2452d9-1730-458c-b380-dbcfd9e5be19-kube-api-access-kxjjz\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:50.914305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913668 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c35b9597-006b-4b14-97e5-9e7d06ed8cd4-sys-fs\") pod \"aws-ebs-csi-driver-node-9cgn8\" (UID: \"c35b9597-006b-4b14-97e5-9e7d06ed8cd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:50.914875 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913692 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fffdl\" (UniqueName: \"kubernetes.io/projected/7a24be49-54d5-46bf-b7ed-73b43b0bcb54-kube-api-access-fffdl\") pod \"node-resolver-q25fm\" (UID: \"7a24be49-54d5-46bf-b7ed-73b43b0bcb54\") " pod="openshift-dns/node-resolver-q25fm" Sep 11 14:14:50.914875 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913719 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5159b394-744e-49c1-932d-a78367218c21-serviceca\") pod \"node-ca-bn8wp\" (UID: \"5159b394-744e-49c1-932d-a78367218c21\") " pod="openshift-image-registry/node-ca-bn8wp" Sep 11 14:14:50.914875 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913745 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdd8n\" (UniqueName: \"kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n\") pod \"network-check-target-jgdkq\" (UID: \"7a80c038-1fe1-4564-875f-e64fa2c5b2a6\") " pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:14:50.914875 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913773 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1cc0d3d9-09b0-48f9-99b7-2a11488eed7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8jm26\" (UID: \"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a\") " pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:50.914875 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.913815 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-var-lib-openvswitch\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:50.914875 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.914046 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5159b394-744e-49c1-932d-a78367218c21-serviceca\") pod \"node-ca-bn8wp\" (UID: \"5159b394-744e-49c1-932d-a78367218c21\") " pod="openshift-image-registry/node-ca-bn8wp" Sep 11 14:14:50.927722 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.927702 2553 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 14:14:50.931105 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.931084 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z94lx\" (UniqueName: \"kubernetes.io/projected/05960fb2-54e8-48b2-852b-caabc3c413ca-kube-api-access-z94lx\") pod \"network-metrics-daemon-s2sfs\" (UID: \"05960fb2-54e8-48b2-852b-caabc3c413ca\") " pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:14:50.935197 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:50.935179 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:14:50.935291 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:50.935200 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:14:50.935291 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:50.935213 2553 projected.go:196] Error preparing data for projected volume kube-api-access-fdd8n for pod openshift-network-diagnostics/network-check-target-jgdkq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:14:50.935291 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:50.935276 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n podName:7a80c038-1fe1-4564-875f-e64fa2c5b2a6 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:51.435258181 +0000 UTC m=+2.049364518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fdd8n" (UniqueName: "kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n") pod "network-check-target-jgdkq" (UID: "7a80c038-1fe1-4564-875f-e64fa2c5b2a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:14:50.937862 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:50.937844 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqz4f\" (UniqueName: \"kubernetes.io/projected/5159b394-744e-49c1-932d-a78367218c21-kube-api-access-tqz4f\") pod \"node-ca-bn8wp\" (UID: \"5159b394-744e-49c1-932d-a78367218c21\") " pod="openshift-image-registry/node-ca-bn8wp" Sep 11 14:14:51.015010 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.014980 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-node-log\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.015148 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015017 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.015148 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015037 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b2452d9-1730-458c-b380-dbcfd9e5be19-ovn-node-metrics-cert\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.015148 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015088 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.015148 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015116 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3ef9965-9ec2-465d-b450-84ba57947908-host-slash\") pod \"iptables-alerter-ldh4b\" (UID: \"f3ef9965-9ec2-465d-b450-84ba57947908\") " pod="openshift-network-operator/iptables-alerter-ldh4b" Sep 11 14:14:51.015148 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015110 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-node-log\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.015352 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015152 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c35b9597-006b-4b14-97e5-9e7d06ed8cd4-socket-dir\") pod \"aws-ebs-csi-driver-node-9cgn8\" (UID: \"c35b9597-006b-4b14-97e5-9e7d06ed8cd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:51.015352 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015180 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-etc-sysctl-conf\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.015352 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015202 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-run\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.015352 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015178 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3ef9965-9ec2-465d-b450-84ba57947908-host-slash\") pod \"iptables-alerter-ldh4b\" (UID: \"f3ef9965-9ec2-465d-b450-84ba57947908\") " pod="openshift-network-operator/iptables-alerter-ldh4b" Sep 11 14:14:51.015352 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015226 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1cc0d3d9-09b0-48f9-99b7-2a11488eed7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8jm26\" (UID: \"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a\") " pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:51.015352 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015290 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c35b9597-006b-4b14-97e5-9e7d06ed8cd4-socket-dir\") pod \"aws-ebs-csi-driver-node-9cgn8\" (UID: \"c35b9597-006b-4b14-97e5-9e7d06ed8cd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:51.015654 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015452 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-etc-sysctl-conf\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.015654 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015486 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-run\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.015654 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015515 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-run-ovn\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.015654 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015545 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-etc-systemd\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.015654 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015577 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-var-lib-kubelet\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.015654 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015611 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed3f15f3-e064-4e2e-972b-07e824971be7-cni-binary-copy\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.015654 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015612 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-run-ovn\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.015654 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015635 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-host-run-netns\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.016133 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015659 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlcfx\" (UniqueName: \"kubernetes.io/projected/ed3f15f3-e064-4e2e-972b-07e824971be7-kube-api-access-dlcfx\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.016133 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015678 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-var-lib-kubelet\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.016133 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015689 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-run-openvswitch\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.016133 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015711 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b2452d9-1730-458c-b380-dbcfd9e5be19-ovnkube-config\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.016133 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015731 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-etc-systemd\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.016133 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015739 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b2452d9-1730-458c-b380-dbcfd9e5be19-ovnkube-script-lib\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.016133 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015779 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-host-run-multus-certs\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.016133 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015805 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6288df4f-7888-432f-b13f-463cb6d8cdd0-tmp\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.016133 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015862 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-host-run-k8s-cni-cncf-io\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.016133 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015887 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-host-var-lib-cni-bin\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.016133 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015909 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-multus-conf-dir\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.016133 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015927 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1cc0d3d9-09b0-48f9-99b7-2a11488eed7a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8jm26\" (UID: \"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a\") " pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:51.016133 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015936 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9df8w\" (UniqueName: \"kubernetes.io/projected/c35b9597-006b-4b14-97e5-9e7d06ed8cd4-kube-api-access-9df8w\") pod \"aws-ebs-csi-driver-node-9cgn8\" (UID: \"c35b9597-006b-4b14-97e5-9e7d06ed8cd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:51.016133 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015963 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbb4cb01-8e27-4823-9410-d5b11334c3a9-dbus\") pod \"global-pull-secret-syncer-n7x6t\" (UID: \"bbb4cb01-8e27-4823-9410-d5b11334c3a9\") " pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:14:51.016133 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015986 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-multus-socket-dir-parent\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.016133 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.015993 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-host-run-netns\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.016133 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016010 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-etc-openvswitch\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.017202 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016001 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-host-run-multus-certs\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.017202 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016046 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-multus-conf-dir\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.017202 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016088 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-run-openvswitch\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.017202 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016400 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bbb4cb01-8e27-4823-9410-d5b11334c3a9-dbus\") pod \"global-pull-secret-syncer-n7x6t\" (UID: \"bbb4cb01-8e27-4823-9410-d5b11334c3a9\") " pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:14:51.017202 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016467 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-multus-socket-dir-parent\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.017202 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016524 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-etc-openvswitch\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.017202 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016631 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b2452d9-1730-458c-b380-dbcfd9e5be19-ovnkube-config\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.017202 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016647 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b2452d9-1730-458c-b380-dbcfd9e5be19-ovnkube-script-lib\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.017202 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016705 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-log-socket\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.017202 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016724 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-host-var-lib-cni-bin\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.017202 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016734 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-host-run-k8s-cni-cncf-io\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.017202 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016739 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c35b9597-006b-4b14-97e5-9e7d06ed8cd4-etc-selinux\") pod \"aws-ebs-csi-driver-node-9cgn8\" (UID: \"c35b9597-006b-4b14-97e5-9e7d06ed8cd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:51.017202 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016769 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-log-socket\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.017202 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016783 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-lib-modules\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.017202 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016809 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-multus-cni-dir\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.017202 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016824 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c35b9597-006b-4b14-97e5-9e7d06ed8cd4-etc-selinux\") pod \"aws-ebs-csi-driver-node-9cgn8\" (UID: \"c35b9597-006b-4b14-97e5-9e7d06ed8cd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:51.017202 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016830 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed3f15f3-e064-4e2e-972b-07e824971be7-multus-daemon-config\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.017202 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016842 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ed3f15f3-e064-4e2e-972b-07e824971be7-cni-binary-copy\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.018165 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016863 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret\") pod \"global-pull-secret-syncer-n7x6t\" (UID: \"bbb4cb01-8e27-4823-9410-d5b11334c3a9\") " pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:14:51.018165 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016867 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-lib-modules\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.018165 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016893 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-etc-kubernetes\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.018165 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016919 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-host-slash\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.018165 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016944 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1cc0d3d9-09b0-48f9-99b7-2a11488eed7a-cnibin\") pod \"multus-additional-cni-plugins-8jm26\" (UID: \"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a\") " pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:51.018165 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:51.016952 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:14:51.018165 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016969 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-host-run-netns\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.018165 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016991 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-etc-kubernetes\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.018165 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016996 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-host-run-ovn-kubernetes\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.018165 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:51.017010 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret podName:bbb4cb01-8e27-4823-9410-d5b11334c3a9 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:51.516991135 +0000 UTC m=+2.131097480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret") pod "global-pull-secret-syncer-n7x6t" (UID: "bbb4cb01-8e27-4823-9410-d5b11334c3a9") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:14:51.018165 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017031 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-host-run-ovn-kubernetes\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.018165 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017042 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-host-slash\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.018165 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017076 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-host-run-netns\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.018165 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017081 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1cc0d3d9-09b0-48f9-99b7-2a11488eed7a-cnibin\") pod \"multus-additional-cni-plugins-8jm26\" (UID: \"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a\") " pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:51.018165 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.016919 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-multus-cni-dir\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.018165 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017124 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b1166baf-c7df-4e63-9360-81dec348a479-agent-certs\") pod \"konnectivity-agent-4zsn9\" (UID: \"b1166baf-c7df-4e63-9360-81dec348a479\") " pod="kube-system/konnectivity-agent-4zsn9" Sep 11 14:14:51.018165 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017148 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1cc0d3d9-09b0-48f9-99b7-2a11488eed7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-8jm26\" (UID: \"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a\") " pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:51.019062 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017172 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1cc0d3d9-09b0-48f9-99b7-2a11488eed7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8jm26\" (UID: \"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a\") " pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:51.019062 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017195 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-cnibin\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.019062 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017216 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-hostroot\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.019062 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017238 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-etc-kubernetes\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.019062 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017262 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxjjz\" (UniqueName: \"kubernetes.io/projected/6b2452d9-1730-458c-b380-dbcfd9e5be19-kube-api-access-kxjjz\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.019062 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017288 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c35b9597-006b-4b14-97e5-9e7d06ed8cd4-sys-fs\") pod \"aws-ebs-csi-driver-node-9cgn8\" (UID: \"c35b9597-006b-4b14-97e5-9e7d06ed8cd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:51.019062 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017310 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fffdl\" (UniqueName: \"kubernetes.io/projected/7a24be49-54d5-46bf-b7ed-73b43b0bcb54-kube-api-access-fffdl\") pod \"node-resolver-q25fm\" (UID: \"7a24be49-54d5-46bf-b7ed-73b43b0bcb54\") " pod="openshift-dns/node-resolver-q25fm" Sep 11 14:14:51.019062 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017333 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ed3f15f3-e064-4e2e-972b-07e824971be7-multus-daemon-config\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.019062 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017382 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1cc0d3d9-09b0-48f9-99b7-2a11488eed7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8jm26\" (UID: \"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a\") " pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:51.019062 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017438 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-var-lib-openvswitch\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.019062 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017470 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbb4cb01-8e27-4823-9410-d5b11334c3a9-kubelet-config\") pod \"global-pull-secret-syncer-n7x6t\" (UID: \"bbb4cb01-8e27-4823-9410-d5b11334c3a9\") " pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:14:51.019062 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017514 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-host-kubelet\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.019062 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017543 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f3ef9965-9ec2-465d-b450-84ba57947908-iptables-alerter-script\") pod \"iptables-alerter-ldh4b\" (UID: \"f3ef9965-9ec2-465d-b450-84ba57947908\") " pod="openshift-network-operator/iptables-alerter-ldh4b" Sep 11 14:14:51.019062 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017572 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqpkt\" (UniqueName: \"kubernetes.io/projected/f3ef9965-9ec2-465d-b450-84ba57947908-kube-api-access-nqpkt\") pod \"iptables-alerter-ldh4b\" (UID: \"f3ef9965-9ec2-465d-b450-84ba57947908\") " pod="openshift-network-operator/iptables-alerter-ldh4b" Sep 11 14:14:51.019062 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017598 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c35b9597-006b-4b14-97e5-9e7d06ed8cd4-device-dir\") pod \"aws-ebs-csi-driver-node-9cgn8\" (UID: \"c35b9597-006b-4b14-97e5-9e7d06ed8cd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:51.019062 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017631 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7a24be49-54d5-46bf-b7ed-73b43b0bcb54-hosts-file\") pod \"node-resolver-q25fm\" (UID: \"7a24be49-54d5-46bf-b7ed-73b43b0bcb54\") " pod="openshift-dns/node-resolver-q25fm" Sep 11 14:14:51.019062 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017661 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1cc0d3d9-09b0-48f9-99b7-2a11488eed7a-cni-binary-copy\") pod \"multus-additional-cni-plugins-8jm26\" (UID: \"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a\") " pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:51.019967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017820 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1cc0d3d9-09b0-48f9-99b7-2a11488eed7a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8jm26\" (UID: \"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a\") " pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:51.019967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017864 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-var-lib-openvswitch\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.019967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017914 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bbb4cb01-8e27-4823-9410-d5b11334c3a9-kubelet-config\") pod \"global-pull-secret-syncer-n7x6t\" (UID: \"bbb4cb01-8e27-4823-9410-d5b11334c3a9\") " pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:14:51.019967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017951 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a24be49-54d5-46bf-b7ed-73b43b0bcb54-tmp-dir\") pod \"node-resolver-q25fm\" (UID: \"7a24be49-54d5-46bf-b7ed-73b43b0bcb54\") " pod="openshift-dns/node-resolver-q25fm" Sep 11 14:14:51.019967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017954 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-host-kubelet\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.019967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.017664 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a24be49-54d5-46bf-b7ed-73b43b0bcb54-tmp-dir\") pod \"node-resolver-q25fm\" (UID: \"7a24be49-54d5-46bf-b7ed-73b43b0bcb54\") " pod="openshift-dns/node-resolver-q25fm" Sep 11 14:14:51.019967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018012 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-etc-sysconfig\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.019967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018041 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-host\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.019967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018069 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkr7m\" (UniqueName: \"kubernetes.io/projected/1cc0d3d9-09b0-48f9-99b7-2a11488eed7a-kube-api-access-gkr7m\") pod \"multus-additional-cni-plugins-8jm26\" (UID: \"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a\") " pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:51.019967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018075 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1cc0d3d9-09b0-48f9-99b7-2a11488eed7a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8jm26\" (UID: \"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a\") " pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:51.019967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018095 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c35b9597-006b-4b14-97e5-9e7d06ed8cd4-registration-dir\") pod \"aws-ebs-csi-driver-node-9cgn8\" (UID: \"c35b9597-006b-4b14-97e5-9e7d06ed8cd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:51.019967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018136 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-sys\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.019967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018141 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-cnibin\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.019967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018161 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b1166baf-c7df-4e63-9360-81dec348a479-konnectivity-ca\") pod \"konnectivity-agent-4zsn9\" (UID: \"b1166baf-c7df-4e63-9360-81dec348a479\") " pod="kube-system/konnectivity-agent-4zsn9" Sep 11 14:14:51.019967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018179 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-hostroot\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.019967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018186 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c35b9597-006b-4b14-97e5-9e7d06ed8cd4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9cgn8\" (UID: \"c35b9597-006b-4b14-97e5-9e7d06ed8cd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:51.019967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018220 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-etc-kubernetes\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.019967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018231 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6288df4f-7888-432f-b13f-463cb6d8cdd0-etc-tuned\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.020672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018258 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-host-cni-bin\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.020672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018281 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-host-cni-netd\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.020672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018308 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b2452d9-1730-458c-b380-dbcfd9e5be19-env-overrides\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.020672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018333 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-etc-sysctl-d\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.020672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018359 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-systemd-units\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.020672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018385 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-etc-modprobe-d\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.020672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018414 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm8kj\" (UniqueName: \"kubernetes.io/projected/6288df4f-7888-432f-b13f-463cb6d8cdd0-kube-api-access-xm8kj\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.020672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018440 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-system-cni-dir\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.020672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018457 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f3ef9965-9ec2-465d-b450-84ba57947908-iptables-alerter-script\") pod \"iptables-alerter-ldh4b\" (UID: \"f3ef9965-9ec2-465d-b450-84ba57947908\") " pod="openshift-network-operator/iptables-alerter-ldh4b" Sep 11 14:14:51.020672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018465 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-host-var-lib-cni-multus\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.020672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018482 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c35b9597-006b-4b14-97e5-9e7d06ed8cd4-sys-fs\") pod \"aws-ebs-csi-driver-node-9cgn8\" (UID: \"c35b9597-006b-4b14-97e5-9e7d06ed8cd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:51.020672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018509 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-host-var-lib-kubelet\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.020672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018541 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-run-systemd\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.020672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018566 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1cc0d3d9-09b0-48f9-99b7-2a11488eed7a-system-cni-dir\") pod \"multus-additional-cni-plugins-8jm26\" (UID: \"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a\") " pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:51.020672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018562 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-host-cni-bin\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.020672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018654 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-host-cni-netd\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.020672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018704 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c35b9597-006b-4b14-97e5-9e7d06ed8cd4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9cgn8\" (UID: \"c35b9597-006b-4b14-97e5-9e7d06ed8cd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:51.021290 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018732 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c35b9597-006b-4b14-97e5-9e7d06ed8cd4-device-dir\") pod \"aws-ebs-csi-driver-node-9cgn8\" (UID: \"c35b9597-006b-4b14-97e5-9e7d06ed8cd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:51.021290 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018747 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-sys\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.021290 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018789 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7a24be49-54d5-46bf-b7ed-73b43b0bcb54-hosts-file\") pod \"node-resolver-q25fm\" (UID: \"7a24be49-54d5-46bf-b7ed-73b43b0bcb54\") " pod="openshift-dns/node-resolver-q25fm" Sep 11 14:14:51.021290 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.018912 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b2452d9-1730-458c-b380-dbcfd9e5be19-ovn-node-metrics-cert\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.021290 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.019119 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b2452d9-1730-458c-b380-dbcfd9e5be19-env-overrides\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.021290 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.019179 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-host-var-lib-kubelet\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.021290 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.019272 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-etc-sysctl-d\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.021290 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.019311 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-systemd-units\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.021290 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.019333 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b1166baf-c7df-4e63-9360-81dec348a479-konnectivity-ca\") pod \"konnectivity-agent-4zsn9\" (UID: \"b1166baf-c7df-4e63-9360-81dec348a479\") " pod="kube-system/konnectivity-agent-4zsn9" Sep 11 14:14:51.021290 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.019360 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c35b9597-006b-4b14-97e5-9e7d06ed8cd4-registration-dir\") pod \"aws-ebs-csi-driver-node-9cgn8\" (UID: \"c35b9597-006b-4b14-97e5-9e7d06ed8cd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:51.021290 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.019370 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-host\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.021290 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.019389 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-system-cni-dir\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.021290 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.019419 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b2452d9-1730-458c-b380-dbcfd9e5be19-run-systemd\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.021290 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.019468 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1cc0d3d9-09b0-48f9-99b7-2a11488eed7a-system-cni-dir\") pod \"multus-additional-cni-plugins-8jm26\" (UID: \"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a\") " pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:51.021290 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.019469 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-etc-modprobe-d\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.021290 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.019540 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ed3f15f3-e064-4e2e-972b-07e824971be7-host-var-lib-cni-multus\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.021290 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.019562 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6288df4f-7888-432f-b13f-463cb6d8cdd0-etc-sysconfig\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.021290 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.020506 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6288df4f-7888-432f-b13f-463cb6d8cdd0-tmp\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.021884 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.020941 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6288df4f-7888-432f-b13f-463cb6d8cdd0-etc-tuned\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.021884 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.021094 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b1166baf-c7df-4e63-9360-81dec348a479-agent-certs\") pod \"konnectivity-agent-4zsn9\" (UID: \"b1166baf-c7df-4e63-9360-81dec348a479\") " pod="kube-system/konnectivity-agent-4zsn9" Sep 11 14:14:51.022437 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:51.022413 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca2a8aca32a15dbbbb325138183d766.slice/crio-f9cd7e91388b9bc8de6c83e0bc4c3425968f316c4bf81df6f615951c45821eb5 WatchSource:0}: Error finding container f9cd7e91388b9bc8de6c83e0bc4c3425968f316c4bf81df6f615951c45821eb5: Status 404 returned error can't find the container with id f9cd7e91388b9bc8de6c83e0bc4c3425968f316c4bf81df6f615951c45821eb5 Sep 11 14:14:51.023110 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:51.023095 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c73b8b4f6cacbe91fc9321bf2769db6.slice/crio-d0b932b5ddfb62e78a8c8414ca95c651e4458aee936e8c890591fd52a2e7800d WatchSource:0}: Error finding container d0b932b5ddfb62e78a8c8414ca95c651e4458aee936e8c890591fd52a2e7800d: Status 404 returned error can't find the container with id d0b932b5ddfb62e78a8c8414ca95c651e4458aee936e8c890591fd52a2e7800d Sep 11 14:14:51.027952 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.027928 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:14:51.107437 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.107403 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9df8w\" (UniqueName: \"kubernetes.io/projected/c35b9597-006b-4b14-97e5-9e7d06ed8cd4-kube-api-access-9df8w\") pod \"aws-ebs-csi-driver-node-9cgn8\" (UID: \"c35b9597-006b-4b14-97e5-9e7d06ed8cd4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:51.108443 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.108424 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkr7m\" (UniqueName: \"kubernetes.io/projected/1cc0d3d9-09b0-48f9-99b7-2a11488eed7a-kube-api-access-gkr7m\") pod \"multus-additional-cni-plugins-8jm26\" (UID: \"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a\") " pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:51.112198 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.112170 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqpkt\" (UniqueName: \"kubernetes.io/projected/f3ef9965-9ec2-465d-b450-84ba57947908-kube-api-access-nqpkt\") pod \"iptables-alerter-ldh4b\" (UID: \"f3ef9965-9ec2-465d-b450-84ba57947908\") " pod="openshift-network-operator/iptables-alerter-ldh4b" Sep 11 14:14:51.116931 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.116877 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fffdl\" (UniqueName: \"kubernetes.io/projected/7a24be49-54d5-46bf-b7ed-73b43b0bcb54-kube-api-access-fffdl\") pod \"node-resolver-q25fm\" (UID: \"7a24be49-54d5-46bf-b7ed-73b43b0bcb54\") " pod="openshift-dns/node-resolver-q25fm" Sep 11 14:14:51.117048 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.117029 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxjjz\" (UniqueName: \"kubernetes.io/projected/6b2452d9-1730-458c-b380-dbcfd9e5be19-kube-api-access-kxjjz\") pod \"ovnkube-node-mmqjh\" (UID: \"6b2452d9-1730-458c-b380-dbcfd9e5be19\") " pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.137682 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.137650 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bn8wp" Sep 11 14:14:51.149026 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:51.148992 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5159b394_744e_49c1_932d_a78367218c21.slice/crio-c3e02160d0eafde2296f966159f7a7af28168001db0a6194f97b8ffa1534ffdc WatchSource:0}: Error finding container c3e02160d0eafde2296f966159f7a7af28168001db0a6194f97b8ffa1534ffdc: Status 404 returned error can't find the container with id c3e02160d0eafde2296f966159f7a7af28168001db0a6194f97b8ffa1534ffdc Sep 11 14:14:51.149439 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.149414 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm8kj\" (UniqueName: \"kubernetes.io/projected/6288df4f-7888-432f-b13f-463cb6d8cdd0-kube-api-access-xm8kj\") pod \"tuned-t89f2\" (UID: \"6288df4f-7888-432f-b13f-463cb6d8cdd0\") " pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.149866 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.149841 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlcfx\" (UniqueName: \"kubernetes.io/projected/ed3f15f3-e064-4e2e-972b-07e824971be7-kube-api-access-dlcfx\") pod \"multus-t5h5f\" (UID: \"ed3f15f3-e064-4e2e-972b-07e824971be7\") " pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.152602 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.152582 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ldh4b" Sep 11 14:14:51.161892 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:51.161868 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ef9965_9ec2_465d_b450_84ba57947908.slice/crio-038cb49bbe1cbee03826e1f0704e46dbfd42a2314cfd48953ba6ef54e6024dd6 WatchSource:0}: Error finding container 038cb49bbe1cbee03826e1f0704e46dbfd42a2314cfd48953ba6ef54e6024dd6: Status 404 returned error can't find the container with id 038cb49bbe1cbee03826e1f0704e46dbfd42a2314cfd48953ba6ef54e6024dd6 Sep 11 14:14:51.166186 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.166164 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4zsn9" Sep 11 14:14:51.174363 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:51.174341 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1166baf_c7df_4e63_9360_81dec348a479.slice/crio-2374d0fbc3e6e138d79dc19cb33689d2d353e0274aa91e6c3f9fb2dcffa37374 WatchSource:0}: Error finding container 2374d0fbc3e6e138d79dc19cb33689d2d353e0274aa91e6c3f9fb2dcffa37374: Status 404 returned error can't find the container with id 2374d0fbc3e6e138d79dc19cb33689d2d353e0274aa91e6c3f9fb2dcffa37374 Sep 11 14:14:51.187425 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.187404 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" Sep 11 14:14:51.197584 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:51.197560 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35b9597_006b_4b14_97e5_9e7d06ed8cd4.slice/crio-b8808e7590868deba32f84a789531b4441fdfae496eedc32eff3c3046df57f6b WatchSource:0}: Error finding container b8808e7590868deba32f84a789531b4441fdfae496eedc32eff3c3046df57f6b: Status 404 returned error can't find the container with id b8808e7590868deba32f84a789531b4441fdfae496eedc32eff3c3046df57f6b Sep 11 14:14:51.200905 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.200885 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t89f2" Sep 11 14:14:51.209240 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:51.209220 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6288df4f_7888_432f_b13f_463cb6d8cdd0.slice/crio-d9a1279084d1d966fe6e41a7c3c341ded2079dbe5adfc64098590c2136c55895 WatchSource:0}: Error finding container d9a1279084d1d966fe6e41a7c3c341ded2079dbe5adfc64098590c2136c55895: Status 404 returned error can't find the container with id d9a1279084d1d966fe6e41a7c3c341ded2079dbe5adfc64098590c2136c55895 Sep 11 14:14:51.217921 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.217900 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8jm26" Sep 11 14:14:51.227616 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:51.227593 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc0d3d9_09b0_48f9_99b7_2a11488eed7a.slice/crio-6672e2003178c0cf58fc27299410c8bfc4c45764a96dba04f40dd03f36a8ccbb WatchSource:0}: Error finding container 6672e2003178c0cf58fc27299410c8bfc4c45764a96dba04f40dd03f36a8ccbb: Status 404 returned error can't find the container with id 6672e2003178c0cf58fc27299410c8bfc4c45764a96dba04f40dd03f36a8ccbb Sep 11 14:14:51.231696 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.231675 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t5h5f" Sep 11 14:14:51.239816 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:51.239772 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3f15f3_e064_4e2e_972b_07e824971be7.slice/crio-7b49d6b2687a14c0049000e5f596478b4663ee14d1f8e967d7a6954e5ad6cd05 WatchSource:0}: Error finding container 7b49d6b2687a14c0049000e5f596478b4663ee14d1f8e967d7a6954e5ad6cd05: Status 404 returned error can't find the container with id 7b49d6b2687a14c0049000e5f596478b4663ee14d1f8e967d7a6954e5ad6cd05 Sep 11 14:14:51.250952 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.250931 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:14:51.255803 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.255786 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q25fm" Sep 11 14:14:51.260287 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:51.260264 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2452d9_1730_458c_b380_dbcfd9e5be19.slice/crio-29810c3b9d8f6243ded2186eb3da4a112ca0ca518b1959597a6d6be07da1dfe2 WatchSource:0}: Error finding container 29810c3b9d8f6243ded2186eb3da4a112ca0ca518b1959597a6d6be07da1dfe2: Status 404 returned error can't find the container with id 29810c3b9d8f6243ded2186eb3da4a112ca0ca518b1959597a6d6be07da1dfe2 Sep 11 14:14:51.264943 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:14:51.264795 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a24be49_54d5_46bf_b7ed_73b43b0bcb54.slice/crio-bc75c1e52f8fcd3286b85506b0f6d5d438f6b4cdd62e40a8c587cab51d16a1fa WatchSource:0}: Error finding container bc75c1e52f8fcd3286b85506b0f6d5d438f6b4cdd62e40a8c587cab51d16a1fa: Status 404 returned error can't find the container with id bc75c1e52f8fcd3286b85506b0f6d5d438f6b4cdd62e40a8c587cab51d16a1fa Sep 11 14:14:51.368557 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.368480 2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:14:51.421604 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.421570 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs\") pod \"network-metrics-daemon-s2sfs\" (UID: \"05960fb2-54e8-48b2-852b-caabc3c413ca\") " pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:14:51.421777 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:51.421687 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:14:51.421777 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:51.421738 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs podName:05960fb2-54e8-48b2-852b-caabc3c413ca nodeName:}" failed. No retries permitted until 2026-09-11 14:14:52.421725723 +0000 UTC m=+3.035832052 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs") pod "network-metrics-daemon-s2sfs" (UID: "05960fb2-54e8-48b2-852b-caabc3c413ca") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:14:51.522411 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.522369 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret\") pod \"global-pull-secret-syncer-n7x6t\" (UID: \"bbb4cb01-8e27-4823-9410-d5b11334c3a9\") " pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:14:51.522629 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.522433 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdd8n\" (UniqueName: \"kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n\") pod \"network-check-target-jgdkq\" (UID: \"7a80c038-1fe1-4564-875f-e64fa2c5b2a6\") " pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:14:51.522629 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:51.522550 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:14:51.522629 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:51.522617 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:14:51.522810 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:51.522637 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:14:51.522810 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:51.522640 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret podName:bbb4cb01-8e27-4823-9410-d5b11334c3a9 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:52.522616492 +0000 UTC m=+3.136722818 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret") pod "global-pull-secret-syncer-n7x6t" (UID: "bbb4cb01-8e27-4823-9410-d5b11334c3a9") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:14:51.522810 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:51.522653 2553 projected.go:196] Error preparing data for projected volume kube-api-access-fdd8n for pod openshift-network-diagnostics/network-check-target-jgdkq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:14:51.522810 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:51.522701 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n podName:7a80c038-1fe1-4564-875f-e64fa2c5b2a6 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:52.522688647 +0000 UTC m=+3.136794970 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fdd8n" (UniqueName: "kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n") pod "network-check-target-jgdkq" (UID: "7a80c038-1fe1-4564-875f-e64fa2c5b2a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:14:51.700193 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.700107 2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:14:51.870466 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.870303 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 14:09:51 +0000 UTC" deadline="2028-06-15 21:37:58.263479826 +0000 UTC" Sep 11 14:14:51.870466 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.870354 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15439h23m6.393131402s" Sep 11 14:14:51.931349 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.931059 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bn8wp" event={"ID":"5159b394-744e-49c1-932d-a78367218c21","Type":"ContainerStarted","Data":"c3e02160d0eafde2296f966159f7a7af28168001db0a6194f97b8ffa1534ffdc"} Sep 11 14:14:51.946134 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.946045 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-17.ec2.internal" event={"ID":"7c73b8b4f6cacbe91fc9321bf2769db6","Type":"ContainerStarted","Data":"d0b932b5ddfb62e78a8c8414ca95c651e4458aee936e8c890591fd52a2e7800d"} Sep 11 14:14:51.949464 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.949433 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8jm26" event={"ID":"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a","Type":"ContainerStarted","Data":"6672e2003178c0cf58fc27299410c8bfc4c45764a96dba04f40dd03f36a8ccbb"} Sep 11 14:14:51.952312 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.952154 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t89f2" event={"ID":"6288df4f-7888-432f-b13f-463cb6d8cdd0","Type":"ContainerStarted","Data":"d9a1279084d1d966fe6e41a7c3c341ded2079dbe5adfc64098590c2136c55895"} Sep 11 14:14:51.984575 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:51.984511 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-17.ec2.internal" event={"ID":"eca2a8aca32a15dbbbb325138183d766","Type":"ContainerStarted","Data":"f9cd7e91388b9bc8de6c83e0bc4c3425968f316c4bf81df6f615951c45821eb5"} Sep 11 14:14:52.012770 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:52.012688 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q25fm" event={"ID":"7a24be49-54d5-46bf-b7ed-73b43b0bcb54","Type":"ContainerStarted","Data":"bc75c1e52f8fcd3286b85506b0f6d5d438f6b4cdd62e40a8c587cab51d16a1fa"} Sep 11 14:14:52.014603 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:52.014480 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" event={"ID":"6b2452d9-1730-458c-b380-dbcfd9e5be19","Type":"ContainerStarted","Data":"29810c3b9d8f6243ded2186eb3da4a112ca0ca518b1959597a6d6be07da1dfe2"} Sep 11 14:14:52.029208 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:52.029144 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t5h5f" event={"ID":"ed3f15f3-e064-4e2e-972b-07e824971be7","Type":"ContainerStarted","Data":"7b49d6b2687a14c0049000e5f596478b4663ee14d1f8e967d7a6954e5ad6cd05"} Sep 11 14:14:52.051920 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:52.051831 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" event={"ID":"c35b9597-006b-4b14-97e5-9e7d06ed8cd4","Type":"ContainerStarted","Data":"b8808e7590868deba32f84a789531b4441fdfae496eedc32eff3c3046df57f6b"} Sep 11 14:14:52.079596 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:52.079536 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4zsn9" event={"ID":"b1166baf-c7df-4e63-9360-81dec348a479","Type":"ContainerStarted","Data":"2374d0fbc3e6e138d79dc19cb33689d2d353e0274aa91e6c3f9fb2dcffa37374"} Sep 11 14:14:52.097446 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:52.097408 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ldh4b" event={"ID":"f3ef9965-9ec2-465d-b450-84ba57947908","Type":"ContainerStarted","Data":"038cb49bbe1cbee03826e1f0704e46dbfd42a2314cfd48953ba6ef54e6024dd6"} Sep 11 14:14:52.431002 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:52.430883 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs\") pod \"network-metrics-daemon-s2sfs\" (UID: \"05960fb2-54e8-48b2-852b-caabc3c413ca\") " pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:14:52.431184 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:52.431073 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:14:52.431184 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:52.431141 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs podName:05960fb2-54e8-48b2-852b-caabc3c413ca nodeName:}" failed. No retries permitted until 2026-09-11 14:14:54.431122746 +0000 UTC m=+5.045229088 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs") pod "network-metrics-daemon-s2sfs" (UID: "05960fb2-54e8-48b2-852b-caabc3c413ca") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:14:52.532234 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:52.532195 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret\") pod \"global-pull-secret-syncer-n7x6t\" (UID: \"bbb4cb01-8e27-4823-9410-d5b11334c3a9\") " pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:14:52.532426 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:52.532258 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdd8n\" (UniqueName: \"kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n\") pod \"network-check-target-jgdkq\" (UID: \"7a80c038-1fe1-4564-875f-e64fa2c5b2a6\") " pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:14:52.532522 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:52.532449 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:14:52.532522 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:52.532469 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:14:52.532522 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:52.532481 2553 projected.go:196] Error preparing data for projected volume kube-api-access-fdd8n for pod openshift-network-diagnostics/network-check-target-jgdkq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:14:52.532682 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:52.532551 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n podName:7a80c038-1fe1-4564-875f-e64fa2c5b2a6 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:54.532530972 +0000 UTC m=+5.146637319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fdd8n" (UniqueName: "kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n") pod "network-check-target-jgdkq" (UID: "7a80c038-1fe1-4564-875f-e64fa2c5b2a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:14:52.533004 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:52.532966 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:14:52.533102 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:52.533018 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret podName:bbb4cb01-8e27-4823-9410-d5b11334c3a9 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:54.533003188 +0000 UTC m=+5.147109525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret") pod "global-pull-secret-syncer-n7x6t" (UID: "bbb4cb01-8e27-4823-9410-d5b11334c3a9") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:14:52.871019 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:52.870931 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 14:09:51 +0000 UTC" deadline="2028-02-29 08:48:29.797607118 +0000 UTC" Sep 11 14:14:52.871019 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:52.870973 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12858h33m36.92663882s" Sep 11 14:14:52.906355 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:52.906294 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:14:52.906603 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:52.906443 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7x6t" podUID="bbb4cb01-8e27-4823-9410-d5b11334c3a9" Sep 11 14:14:52.906912 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:52.906864 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:14:52.907009 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:52.906959 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgdkq" podUID="7a80c038-1fe1-4564-875f-e64fa2c5b2a6" Sep 11 14:14:52.907075 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:52.907034 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:14:52.907127 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:52.907116 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2sfs" podUID="05960fb2-54e8-48b2-852b-caabc3c413ca" Sep 11 14:14:53.665346 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:53.665135 2553 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 14:14:54.448241 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:54.448135 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs\") pod \"network-metrics-daemon-s2sfs\" (UID: \"05960fb2-54e8-48b2-852b-caabc3c413ca\") " pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:14:54.448687 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:54.448267 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:14:54.448687 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:54.448327 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs podName:05960fb2-54e8-48b2-852b-caabc3c413ca nodeName:}" failed. No retries permitted until 2026-09-11 14:14:58.448307958 +0000 UTC m=+9.062414282 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs") pod "network-metrics-daemon-s2sfs" (UID: "05960fb2-54e8-48b2-852b-caabc3c413ca") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:14:54.548869 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:54.548830 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret\") pod \"global-pull-secret-syncer-n7x6t\" (UID: \"bbb4cb01-8e27-4823-9410-d5b11334c3a9\") " pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:14:54.549043 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:54.548890 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdd8n\" (UniqueName: \"kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n\") pod \"network-check-target-jgdkq\" (UID: \"7a80c038-1fe1-4564-875f-e64fa2c5b2a6\") " pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:14:54.549099 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:54.549057 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:14:54.549099 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:54.549077 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:14:54.549099 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:54.549089 2553 projected.go:196] Error preparing data for projected volume kube-api-access-fdd8n for pod openshift-network-diagnostics/network-check-target-jgdkq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:14:54.549247 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:54.549143 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n podName:7a80c038-1fe1-4564-875f-e64fa2c5b2a6 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:58.549126588 +0000 UTC m=+9.163232917 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fdd8n" (UniqueName: "kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n") pod "network-check-target-jgdkq" (UID: "7a80c038-1fe1-4564-875f-e64fa2c5b2a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:14:54.549610 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:54.549544 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:14:54.549610 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:54.549597 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret podName:bbb4cb01-8e27-4823-9410-d5b11334c3a9 nodeName:}" failed. No retries permitted until 2026-09-11 14:14:58.549581711 +0000 UTC m=+9.163688039 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret") pod "global-pull-secret-syncer-n7x6t" (UID: "bbb4cb01-8e27-4823-9410-d5b11334c3a9") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:14:54.905995 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:54.905909 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:14:54.906151 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:54.906040 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7x6t" podUID="bbb4cb01-8e27-4823-9410-d5b11334c3a9" Sep 11 14:14:54.906817 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:54.906445 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:14:54.906817 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:54.906540 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgdkq" podUID="7a80c038-1fe1-4564-875f-e64fa2c5b2a6" Sep 11 14:14:54.906817 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:54.906608 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:14:54.906817 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:54.906691 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2sfs" podUID="05960fb2-54e8-48b2-852b-caabc3c413ca" Sep 11 14:14:56.906624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:56.906123 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:14:56.906624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:56.906153 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:14:56.906624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:56.906158 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:14:56.906624 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:56.906260 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7x6t" podUID="bbb4cb01-8e27-4823-9410-d5b11334c3a9" Sep 11 14:14:56.907258 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:56.906643 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2sfs" podUID="05960fb2-54e8-48b2-852b-caabc3c413ca" Sep 11 14:14:56.907258 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:56.906741 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgdkq" podUID="7a80c038-1fe1-4564-875f-e64fa2c5b2a6" Sep 11 14:14:58.482852 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:58.482804 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs\") pod \"network-metrics-daemon-s2sfs\" (UID: \"05960fb2-54e8-48b2-852b-caabc3c413ca\") " pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:14:58.483312 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:58.482924 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:14:58.483312 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:58.482988 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs podName:05960fb2-54e8-48b2-852b-caabc3c413ca nodeName:}" failed. No retries permitted until 2026-09-11 14:15:06.482974393 +0000 UTC m=+17.097080714 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs") pod "network-metrics-daemon-s2sfs" (UID: "05960fb2-54e8-48b2-852b-caabc3c413ca") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:14:58.583725 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:58.583689 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret\") pod \"global-pull-secret-syncer-n7x6t\" (UID: \"bbb4cb01-8e27-4823-9410-d5b11334c3a9\") " pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:14:58.583914 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:58.583744 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdd8n\" (UniqueName: \"kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n\") pod \"network-check-target-jgdkq\" (UID: \"7a80c038-1fe1-4564-875f-e64fa2c5b2a6\") " pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:14:58.583914 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:58.583864 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:14:58.583914 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:58.583883 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:14:58.583914 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:58.583902 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:14:58.583914 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:58.583914 2553 projected.go:196] Error preparing data for projected volume kube-api-access-fdd8n for pod openshift-network-diagnostics/network-check-target-jgdkq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:14:58.584158 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:58.583956 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret podName:bbb4cb01-8e27-4823-9410-d5b11334c3a9 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:06.583929491 +0000 UTC m=+17.198035815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret") pod "global-pull-secret-syncer-n7x6t" (UID: "bbb4cb01-8e27-4823-9410-d5b11334c3a9") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:14:58.584158 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:58.583979 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n podName:7a80c038-1fe1-4564-875f-e64fa2c5b2a6 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:06.583966888 +0000 UTC m=+17.198073214 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fdd8n" (UniqueName: "kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n") pod "network-check-target-jgdkq" (UID: "7a80c038-1fe1-4564-875f-e64fa2c5b2a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:14:58.906486 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:58.905682 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:14:58.906486 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:58.905865 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgdkq" podUID="7a80c038-1fe1-4564-875f-e64fa2c5b2a6" Sep 11 14:14:58.906486 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:58.906232 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:14:58.906486 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:58.906329 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2sfs" podUID="05960fb2-54e8-48b2-852b-caabc3c413ca" Sep 11 14:14:58.906486 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:14:58.906369 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:14:58.906486 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:14:58.906451 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7x6t" podUID="bbb4cb01-8e27-4823-9410-d5b11334c3a9" Sep 11 14:15:00.905848 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:00.905719 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:15:00.905848 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:00.905769 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:15:00.905848 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:00.905772 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:15:00.906382 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:00.905852 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgdkq" podUID="7a80c038-1fe1-4564-875f-e64fa2c5b2a6" Sep 11 14:15:00.906382 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:00.905955 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2sfs" podUID="05960fb2-54e8-48b2-852b-caabc3c413ca" Sep 11 14:15:00.906382 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:00.906039 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7x6t" podUID="bbb4cb01-8e27-4823-9410-d5b11334c3a9" Sep 11 14:15:02.905862 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:02.905835 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:15:02.905862 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:02.905850 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:15:02.905862 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:02.905860 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:15:02.906328 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:02.905953 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgdkq" podUID="7a80c038-1fe1-4564-875f-e64fa2c5b2a6" Sep 11 14:15:02.906328 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:02.905985 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7x6t" podUID="bbb4cb01-8e27-4823-9410-d5b11334c3a9" Sep 11 14:15:02.906328 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:02.906074 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2sfs" podUID="05960fb2-54e8-48b2-852b-caabc3c413ca" Sep 11 14:15:04.905849 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:04.905803 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:15:04.906280 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:04.905803 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:15:04.906280 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:04.905921 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgdkq" podUID="7a80c038-1fe1-4564-875f-e64fa2c5b2a6" Sep 11 14:15:04.906280 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:04.905803 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:15:04.906280 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:04.905978 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7x6t" podUID="bbb4cb01-8e27-4823-9410-d5b11334c3a9" Sep 11 14:15:04.906280 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:04.906061 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2sfs" podUID="05960fb2-54e8-48b2-852b-caabc3c413ca" Sep 11 14:15:06.540590 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:06.540505 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs\") pod \"network-metrics-daemon-s2sfs\" (UID: \"05960fb2-54e8-48b2-852b-caabc3c413ca\") " pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:15:06.541010 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:06.540647 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:06.541010 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:06.540749 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs podName:05960fb2-54e8-48b2-852b-caabc3c413ca nodeName:}" failed. No retries permitted until 2026-09-11 14:15:22.540691719 +0000 UTC m=+33.154798061 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs") pod "network-metrics-daemon-s2sfs" (UID: "05960fb2-54e8-48b2-852b-caabc3c413ca") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:06.641864 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:06.641818 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret\") pod \"global-pull-secret-syncer-n7x6t\" (UID: \"bbb4cb01-8e27-4823-9410-d5b11334c3a9\") " pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:15:06.642051 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:06.641884 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdd8n\" (UniqueName: \"kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n\") pod \"network-check-target-jgdkq\" (UID: \"7a80c038-1fe1-4564-875f-e64fa2c5b2a6\") " pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:15:06.642051 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:06.641959 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:06.642051 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:06.642016 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:15:06.642051 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:06.642031 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret podName:bbb4cb01-8e27-4823-9410-d5b11334c3a9 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:22.642013464 +0000 UTC m=+33.256119806 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret") pod "global-pull-secret-syncer-n7x6t" (UID: "bbb4cb01-8e27-4823-9410-d5b11334c3a9") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:06.642051 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:06.642037 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:15:06.642051 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:06.642050 2553 projected.go:196] Error preparing data for projected volume kube-api-access-fdd8n for pod openshift-network-diagnostics/network-check-target-jgdkq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:06.642314 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:06.642096 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n podName:7a80c038-1fe1-4564-875f-e64fa2c5b2a6 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:22.642080925 +0000 UTC m=+33.256187264 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fdd8n" (UniqueName: "kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n") pod "network-check-target-jgdkq" (UID: "7a80c038-1fe1-4564-875f-e64fa2c5b2a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:06.906141 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:06.906062 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:15:06.906306 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:06.906072 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:15:06.906306 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:06.906196 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7x6t" podUID="bbb4cb01-8e27-4823-9410-d5b11334c3a9" Sep 11 14:15:06.906306 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:06.906291 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2sfs" podUID="05960fb2-54e8-48b2-852b-caabc3c413ca" Sep 11 14:15:06.906481 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:06.906072 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:15:06.906481 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:06.906395 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgdkq" podUID="7a80c038-1fe1-4564-875f-e64fa2c5b2a6" Sep 11 14:15:08.906628 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:08.906589 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:15:08.907084 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:08.906700 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:15:08.907084 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:08.906703 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7x6t" podUID="bbb4cb01-8e27-4823-9410-d5b11334c3a9" Sep 11 14:15:08.907084 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:08.906721 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:15:08.907084 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:08.906804 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgdkq" podUID="7a80c038-1fe1-4564-875f-e64fa2c5b2a6" Sep 11 14:15:08.907084 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:08.906942 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2sfs" podUID="05960fb2-54e8-48b2-852b-caabc3c413ca" Sep 11 14:15:10.906115 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:10.905849 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:15:10.906776 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:10.905946 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:15:10.906776 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:10.906184 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7x6t" podUID="bbb4cb01-8e27-4823-9410-d5b11334c3a9" Sep 11 14:15:10.906776 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:10.906275 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2sfs" podUID="05960fb2-54e8-48b2-852b-caabc3c413ca" Sep 11 14:15:10.906776 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:10.905946 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:15:10.906776 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:10.906347 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgdkq" podUID="7a80c038-1fe1-4564-875f-e64fa2c5b2a6" Sep 11 14:15:11.146528 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.146267 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-17.ec2.internal" event={"ID":"7c73b8b4f6cacbe91fc9321bf2769db6","Type":"ContainerStarted","Data":"c6dced92507b9cf20a5bf722506d58a672167d521d635ef3eeed608aa6cf5915"} Sep 11 14:15:11.147945 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.147909 2553 generic.go:356] "Generic (PLEG): container finished" podID="1cc0d3d9-09b0-48f9-99b7-2a11488eed7a" containerID="8ac79382c725a06c6a61c0f7967b7daa2e18c903824f777579a572e2b2327208" exitCode=0 Sep 11 14:15:11.148050 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.147988 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8jm26" event={"ID":"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a","Type":"ContainerDied","Data":"8ac79382c725a06c6a61c0f7967b7daa2e18c903824f777579a572e2b2327208"} Sep 11 14:15:11.149621 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.149595 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t89f2" event={"ID":"6288df4f-7888-432f-b13f-463cb6d8cdd0","Type":"ContainerStarted","Data":"99a7e6dd355c26d301985edc73af9d6c74bca3d4095ddd01eaed3a24492dcfe1"} Sep 11 14:15:11.151327 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.151303 2553 generic.go:356] "Generic (PLEG): container finished" podID="eca2a8aca32a15dbbbb325138183d766" containerID="028ea5070a70ff2aa1118b96df4249bbbe7bead21fa43c37a9489aaf24832f33" exitCode=0 Sep 11 14:15:11.151440 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.151367 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-17.ec2.internal" event={"ID":"eca2a8aca32a15dbbbb325138183d766","Type":"ContainerDied","Data":"028ea5070a70ff2aa1118b96df4249bbbe7bead21fa43c37a9489aaf24832f33"} Sep 11 14:15:11.153037 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.153017 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q25fm" event={"ID":"7a24be49-54d5-46bf-b7ed-73b43b0bcb54","Type":"ContainerStarted","Data":"a1317b4cdb77a827b60494fa4ebec4cdc93d9340fb3e010304d3ccf883bf8baa"} Sep 11 14:15:11.156379 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.156364 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmqjh_6b2452d9-1730-458c-b380-dbcfd9e5be19/ovn-acl-logging/0.log" Sep 11 14:15:11.156760 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.156741 2553 generic.go:356] "Generic (PLEG): container finished" podID="6b2452d9-1730-458c-b380-dbcfd9e5be19" containerID="579e5942ee56064868161d2ee5d322d930760b722aa9645d7023180b6f42d579" exitCode=1 Sep 11 14:15:11.156822 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.156809 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" event={"ID":"6b2452d9-1730-458c-b380-dbcfd9e5be19","Type":"ContainerStarted","Data":"df5f35dc206e447bd144fb923faa3eb941ff95db7a49e16694ff8147ba102549"} Sep 11 14:15:11.156877 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.156863 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" event={"ID":"6b2452d9-1730-458c-b380-dbcfd9e5be19","Type":"ContainerStarted","Data":"43438fb73d644bf7abeff61a964b1379fee875d7cbe760c7aada3d2a2c78a81c"} Sep 11 14:15:11.156919 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.156903 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" event={"ID":"6b2452d9-1730-458c-b380-dbcfd9e5be19","Type":"ContainerStarted","Data":"d4b442c17128efeb91fde47e7f113be832c1acc353a75ebfdf062a8e9716cd5b"} Sep 11 14:15:11.156962 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.156936 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" event={"ID":"6b2452d9-1730-458c-b380-dbcfd9e5be19","Type":"ContainerStarted","Data":"987a867b5895aa6f712314b21aa6e17d92488b354122f3aa86c2af29d7835816"} Sep 11 14:15:11.156998 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.156964 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" event={"ID":"6b2452d9-1730-458c-b380-dbcfd9e5be19","Type":"ContainerDied","Data":"579e5942ee56064868161d2ee5d322d930760b722aa9645d7023180b6f42d579"} Sep 11 14:15:11.157041 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.157004 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" event={"ID":"6b2452d9-1730-458c-b380-dbcfd9e5be19","Type":"ContainerStarted","Data":"f890774125c8363ee3da8ceec61d9069829bb1f592f54c82c2d19ac13dcf5b5f"} Sep 11 14:15:11.158397 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.158375 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t5h5f" event={"ID":"ed3f15f3-e064-4e2e-972b-07e824971be7","Type":"ContainerStarted","Data":"aac5968f0ff733a70fc4bb536cbf331c55dae42a64a4fb433662b6d190919767"} Sep 11 14:15:11.159864 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.159847 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" event={"ID":"c35b9597-006b-4b14-97e5-9e7d06ed8cd4","Type":"ContainerStarted","Data":"d9a390424f24311fb9a0da24d7a39aefb093935585ecd4c4995ca363b8e49c8b"} Sep 11 14:15:11.161313 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.161288 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4zsn9" event={"ID":"b1166baf-c7df-4e63-9360-81dec348a479","Type":"ContainerStarted","Data":"1e27b1e5ee467fe7ee5a2d45e3eebc0ea2680a2d8f3c86b9f9982852e26adc67"} Sep 11 14:15:11.162679 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.162655 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bn8wp" event={"ID":"5159b394-744e-49c1-932d-a78367218c21","Type":"ContainerStarted","Data":"b6951b6a74f78406eff97cbef263b1a0c8a0bda8e6d000397aece5fbf9481fa8"} Sep 11 14:15:11.224332 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.224273 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-17.ec2.internal" podStartSLOduration=21.224249584 podStartE2EDuration="21.224249584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:11.168181804 +0000 UTC m=+21.782288142" watchObservedRunningTime="2026-09-11 14:15:11.224249584 +0000 UTC m=+21.838355928" Sep 11 14:15:11.268842 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.268793 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-t89f2" podStartSLOduration=2.212697565 podStartE2EDuration="21.26877641s" totalImagesPullingTime="19.056078845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:11.223932191 +0000 UTC m=+21.838038517" watchObservedRunningTime="2026-09-11 14:15:11.26877641 +0000 UTC m=+21.882882753" Sep 11 14:15:11.335303 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.335253 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-t5h5f" podStartSLOduration=2.01052103 podStartE2EDuration="21.335238002s" totalImagesPullingTime="19.324716972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:11.296742122 +0000 UTC m=+21.910848465" watchObservedRunningTime="2026-09-11 14:15:11.335238002 +0000 UTC m=+21.949344345" Sep 11 14:15:11.335530 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.335508 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q25fm" podStartSLOduration=2.349951946 podStartE2EDuration="21.335501364s" totalImagesPullingTime="18.985549418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:11.335334145 +0000 UTC m=+21.949440489" watchObservedRunningTime="2026-09-11 14:15:11.335501364 +0000 UTC m=+21.949607708" Sep 11 14:15:11.374042 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.373992 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bn8wp" podStartSLOduration=2.273127912 podStartE2EDuration="21.373978349s" totalImagesPullingTime="19.100850437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:11.373695131 +0000 UTC m=+21.987801679" watchObservedRunningTime="2026-09-11 14:15:11.373978349 +0000 UTC m=+21.988084693" Sep 11 14:15:11.392505 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:11.392460 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4zsn9" podStartSLOduration=2.301382057 podStartE2EDuration="21.392445856s" totalImagesPullingTime="19.091063799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:11.391969012 +0000 UTC m=+22.006075353" watchObservedRunningTime="2026-09-11 14:15:11.392445856 +0000 UTC m=+22.006552200" Sep 11 14:15:12.166504 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:12.166465 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ldh4b" event={"ID":"f3ef9965-9ec2-465d-b450-84ba57947908","Type":"ContainerStarted","Data":"fab0a02d99ac1bc98aaa0128506f1a9fcc67ba91356ee9f54324dad39e004f4c"} Sep 11 14:15:12.168623 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:12.168555 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-17.ec2.internal" event={"ID":"eca2a8aca32a15dbbbb325138183d766","Type":"ContainerStarted","Data":"85492314bbe97aa57153e5dc3fef9326ac56e541ef37f20de09da4fe19b3ae44"} Sep 11 14:15:12.209801 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:12.209746 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ldh4b" podStartSLOduration=3.106723458 podStartE2EDuration="22.209726772s" totalImagesPullingTime="19.103003314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:12.185939946 +0000 UTC m=+22.800046290" watchObservedRunningTime="2026-09-11 14:15:12.209726772 +0000 UTC m=+22.823833118" Sep 11 14:15:12.209973 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:12.209840 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-17.ec2.internal" podStartSLOduration=22.209836679 podStartE2EDuration="22.209836679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:12.209484394 +0000 UTC m=+22.823590738" watchObservedRunningTime="2026-09-11 14:15:12.209836679 +0000 UTC m=+22.823943024" Sep 11 14:15:12.906172 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:12.906137 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:15:12.906172 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:12.906164 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:15:12.906564 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:12.906164 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:15:12.906564 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:12.906272 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgdkq" podUID="7a80c038-1fe1-4564-875f-e64fa2c5b2a6" Sep 11 14:15:12.906564 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:12.906448 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7x6t" podUID="bbb4cb01-8e27-4823-9410-d5b11334c3a9" Sep 11 14:15:12.906564 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:12.906514 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2sfs" podUID="05960fb2-54e8-48b2-852b-caabc3c413ca" Sep 11 14:15:13.174048 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:13.174025 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmqjh_6b2452d9-1730-458c-b380-dbcfd9e5be19/ovn-acl-logging/0.log" Sep 11 14:15:13.174553 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:13.174530 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" event={"ID":"6b2452d9-1730-458c-b380-dbcfd9e5be19","Type":"ContainerStarted","Data":"20711e5267aaac9c6a8ddc319858c72d0c5a960b3d28118aba9a72143df8e985"} Sep 11 14:15:13.508749 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:13.508725 2553 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 14:15:13.547877 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:13.547849 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4zsn9" Sep 11 14:15:13.633571 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:13.633538 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4zsn9" Sep 11 14:15:13.634831 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:13.634800 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4zsn9" Sep 11 14:15:13.876801 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:13.876629 2553 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T14:15:13.508744537Z","UUID":"1ebf2454-d7c5-482d-9e19-9ab053dfbf2b","Handler":null,"Name":"","Endpoint":""} Sep 11 14:15:13.878764 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:13.878721 2553 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 14:15:13.878764 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:13.878770 2553 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 14:15:14.179120 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:14.179029 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" event={"ID":"c35b9597-006b-4b14-97e5-9e7d06ed8cd4","Type":"ContainerStarted","Data":"1bbe3f3c3c93bf7165a8d4d647bef51fe6cfe6e16d3ec71c91971d4420cbd866"} Sep 11 14:15:14.179828 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:14.179805 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4zsn9" Sep 11 14:15:14.905690 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:14.905658 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:15:14.905690 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:14.905674 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:15:14.905896 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:14.905658 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:15:14.905896 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:14.905784 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7x6t" podUID="bbb4cb01-8e27-4823-9410-d5b11334c3a9" Sep 11 14:15:14.905896 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:14.905864 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2sfs" podUID="05960fb2-54e8-48b2-852b-caabc3c413ca" Sep 11 14:15:14.906044 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:14.905944 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgdkq" podUID="7a80c038-1fe1-4564-875f-e64fa2c5b2a6" Sep 11 14:15:16.184632 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:16.184601 2553 generic.go:356] "Generic (PLEG): container finished" podID="1cc0d3d9-09b0-48f9-99b7-2a11488eed7a" containerID="b1c7cfa440e780ba0ab039d751c06689f754174873fc7e2e0d8d7c4c3ffab11e" exitCode=0 Sep 11 14:15:16.184994 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:16.184680 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8jm26" event={"ID":"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a","Type":"ContainerDied","Data":"b1c7cfa440e780ba0ab039d751c06689f754174873fc7e2e0d8d7c4c3ffab11e"} Sep 11 14:15:16.188235 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:16.188218 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmqjh_6b2452d9-1730-458c-b380-dbcfd9e5be19/ovn-acl-logging/0.log" Sep 11 14:15:16.188566 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:16.188548 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" event={"ID":"6b2452d9-1730-458c-b380-dbcfd9e5be19","Type":"ContainerStarted","Data":"9bc156d07d7031c8644d054fd0011925d18d0e88ba365dee3d1cb9b1e01ab3da"} Sep 11 14:15:16.189023 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:16.189007 2553 scope.go:117] "RemoveContainer" containerID="579e5942ee56064868161d2ee5d322d930760b722aa9645d7023180b6f42d579" Sep 11 14:15:16.906091 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:16.906057 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:15:16.906458 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:16.906060 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:15:16.906458 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:16.906191 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7x6t" podUID="bbb4cb01-8e27-4823-9410-d5b11334c3a9" Sep 11 14:15:16.906458 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:16.906207 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:15:16.906458 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:16.906303 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgdkq" podUID="7a80c038-1fe1-4564-875f-e64fa2c5b2a6" Sep 11 14:15:16.906458 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:16.906406 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2sfs" podUID="05960fb2-54e8-48b2-852b-caabc3c413ca" Sep 11 14:15:17.137272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:17.137245 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:15:17.195101 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:17.195043 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmqjh_6b2452d9-1730-458c-b380-dbcfd9e5be19/ovn-acl-logging/0.log" Sep 11 14:15:17.195710 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:17.195452 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" event={"ID":"6b2452d9-1730-458c-b380-dbcfd9e5be19","Type":"ContainerStarted","Data":"11a7c952703ba296f08f4d2987b3ed45bba20cd6031f4d697341db53084e18a2"} Sep 11 14:15:17.195782 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:17.195747 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:15:17.195839 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:17.195822 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:15:17.198398 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:17.198318 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" event={"ID":"c35b9597-006b-4b14-97e5-9e7d06ed8cd4","Type":"ContainerStarted","Data":"95cafa74f5bf62b899ee39f8a2c9defa4547de1875a8af207e0f2556928ef83b"} Sep 11 14:15:17.212532 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:17.212506 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:15:17.212705 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:17.212689 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:15:17.264524 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:17.264476 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" podStartSLOduration=8.23973433 podStartE2EDuration="27.264457566s" totalImagesPullingTime="19.024723236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:17.251925326 +0000 UTC m=+27.866031653" watchObservedRunningTime="2026-09-11 14:15:17.264457566 +0000 UTC m=+27.878563910" Sep 11 14:15:17.301644 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:17.301596 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9cgn8" podStartSLOduration=3.439829797 podStartE2EDuration="27.301579678s" totalImagesPullingTime="23.861749881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:17.300262768 +0000 UTC m=+27.914369112" watchObservedRunningTime="2026-09-11 14:15:17.301579678 +0000 UTC m=+27.915686022" Sep 11 14:15:17.465461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:17.465429 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s2sfs"] Sep 11 14:15:17.465568 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:17.465553 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:15:17.465702 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:17.465679 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2sfs" podUID="05960fb2-54e8-48b2-852b-caabc3c413ca" Sep 11 14:15:17.466229 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:17.466207 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n7x6t"] Sep 11 14:15:17.466306 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:17.466282 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:15:17.466373 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:17.466357 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7x6t" podUID="bbb4cb01-8e27-4823-9410-d5b11334c3a9" Sep 11 14:15:17.484208 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:17.484184 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jgdkq"] Sep 11 14:15:17.484309 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:17.484260 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:15:17.484344 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:17.484332 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgdkq" podUID="7a80c038-1fe1-4564-875f-e64fa2c5b2a6" Sep 11 14:15:18.202957 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:18.202918 2553 generic.go:356] "Generic (PLEG): container finished" podID="1cc0d3d9-09b0-48f9-99b7-2a11488eed7a" containerID="399fc08770e4c73fdcec33715910822bd3473bd4ab06947041c7a59cbae647d8" exitCode=0 Sep 11 14:15:18.203576 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:18.203004 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8jm26" event={"ID":"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a","Type":"ContainerDied","Data":"399fc08770e4c73fdcec33715910822bd3473bd4ab06947041c7a59cbae647d8"} Sep 11 14:15:18.906225 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:18.906191 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:15:18.906225 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:18.906215 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:15:18.906454 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:18.906323 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:15:18.906454 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:18.906329 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7x6t" podUID="bbb4cb01-8e27-4823-9410-d5b11334c3a9" Sep 11 14:15:18.906532 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:18.906447 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgdkq" podUID="7a80c038-1fe1-4564-875f-e64fa2c5b2a6" Sep 11 14:15:18.906581 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:18.906549 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2sfs" podUID="05960fb2-54e8-48b2-852b-caabc3c413ca" Sep 11 14:15:20.212141 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:20.211835 2553 generic.go:356] "Generic (PLEG): container finished" podID="1cc0d3d9-09b0-48f9-99b7-2a11488eed7a" containerID="34a44c6fa9fdfb3e64fe07ccad29f99df9f5600747d2db88f82dfdf51ba5bdff" exitCode=0 Sep 11 14:15:20.212141 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:20.211901 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8jm26" event={"ID":"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a","Type":"ContainerDied","Data":"34a44c6fa9fdfb3e64fe07ccad29f99df9f5600747d2db88f82dfdf51ba5bdff"} Sep 11 14:15:20.906084 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:20.906047 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:15:20.906268 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:20.906123 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:15:20.906268 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:20.906215 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:15:20.906268 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:20.906220 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7x6t" podUID="bbb4cb01-8e27-4823-9410-d5b11334c3a9" Sep 11 14:15:20.906439 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:20.906321 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2sfs" podUID="05960fb2-54e8-48b2-852b-caabc3c413ca" Sep 11 14:15:20.906439 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:20.906401 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgdkq" podUID="7a80c038-1fe1-4564-875f-e64fa2c5b2a6" Sep 11 14:15:22.556872 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:22.556831 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs\") pod \"network-metrics-daemon-s2sfs\" (UID: \"05960fb2-54e8-48b2-852b-caabc3c413ca\") " pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:15:22.557340 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:22.556996 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:22.557340 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:22.557084 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs podName:05960fb2-54e8-48b2-852b-caabc3c413ca nodeName:}" failed. No retries permitted until 2026-09-11 14:15:54.557063507 +0000 UTC m=+65.171169846 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs") pod "network-metrics-daemon-s2sfs" (UID: "05960fb2-54e8-48b2-852b-caabc3c413ca") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 14:15:22.657380 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:22.657341 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret\") pod \"global-pull-secret-syncer-n7x6t\" (UID: \"bbb4cb01-8e27-4823-9410-d5b11334c3a9\") " pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:15:22.657588 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:22.657403 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdd8n\" (UniqueName: \"kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n\") pod \"network-check-target-jgdkq\" (UID: \"7a80c038-1fe1-4564-875f-e64fa2c5b2a6\") " pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:15:22.657588 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:22.657567 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 14:15:22.657691 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:22.657597 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 14:15:22.657691 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:22.657611 2553 projected.go:196] Error preparing data for projected volume kube-api-access-fdd8n for pod openshift-network-diagnostics/network-check-target-jgdkq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:22.657691 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:22.657577 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:22.657691 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:22.657673 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n podName:7a80c038-1fe1-4564-875f-e64fa2c5b2a6 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:54.657653302 +0000 UTC m=+65.271759628 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fdd8n" (UniqueName: "kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n") pod "network-check-target-jgdkq" (UID: "7a80c038-1fe1-4564-875f-e64fa2c5b2a6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 14:15:22.657691 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:22.657718 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret podName:bbb4cb01-8e27-4823-9410-d5b11334c3a9 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:54.657698705 +0000 UTC m=+65.271805043 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret") pod "global-pull-secret-syncer-n7x6t" (UID: "bbb4cb01-8e27-4823-9410-d5b11334c3a9") : object "kube-system"/"original-pull-secret" not registered Sep 11 14:15:22.906222 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:22.906141 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:15:22.906396 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:22.906256 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:15:22.906396 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:22.906266 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jgdkq" podUID="7a80c038-1fe1-4564-875f-e64fa2c5b2a6" Sep 11 14:15:22.906396 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:22.906309 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:15:22.906576 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:22.906437 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2sfs" podUID="05960fb2-54e8-48b2-852b-caabc3c413ca" Sep 11 14:15:22.906576 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:22.906514 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n7x6t" podUID="bbb4cb01-8e27-4823-9410-d5b11334c3a9" Sep 11 14:15:23.226769 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.226693 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-17.ec2.internal" event="NodeReady" Sep 11 14:15:23.226927 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.226844 2553 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 14:15:23.336957 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.336919 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74c56b68bf-b92lx"] Sep 11 14:15:23.342226 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.342202 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.349344 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.349321 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w4lvw\"" Sep 11 14:15:23.349557 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.349376 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 11 14:15:23.375949 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.375663 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 11 14:15:23.375949 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.375673 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 11 14:15:23.376514 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.376494 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-l5fhb"] Sep 11 14:15:23.381479 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.381450 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l5fhb" Sep 11 14:15:23.389582 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.389538 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 11 14:15:23.390176 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.390152 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74c56b68bf-b92lx"] Sep 11 14:15:23.400482 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.400456 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 14:15:23.400611 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.400497 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 14:15:23.400611 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.400504 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 14:15:23.400824 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.400806 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nv8ln\"" Sep 11 14:15:23.446435 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.446392 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l5fhb"] Sep 11 14:15:23.462804 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.462766 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-certificates\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.462958 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.462828 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c28b757-a450-43a5-8c99-63a0e3824cc8-cert\") pod \"ingress-canary-l5fhb\" (UID: \"3c28b757-a450-43a5-8c99-63a0e3824cc8\") " pod="openshift-ingress-canary/ingress-canary-l5fhb" Sep 11 14:15:23.462958 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.462899 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f32b4da3-04be-4cf9-94b9-03e8ab111b69-installation-pull-secrets\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.462958 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.462928 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f32b4da3-04be-4cf9-94b9-03e8ab111b69-image-registry-private-configuration\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.463122 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.462987 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.463122 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.463038 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f32b4da3-04be-4cf9-94b9-03e8ab111b69-ca-trust-extracted\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.463122 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.463111 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ltmz\" (UniqueName: \"kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-kube-api-access-5ltmz\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.463232 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.463139 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f32b4da3-04be-4cf9-94b9-03e8ab111b69-trusted-ca\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.463232 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.463163 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96kbk\" (UniqueName: \"kubernetes.io/projected/3c28b757-a450-43a5-8c99-63a0e3824cc8-kube-api-access-96kbk\") pod \"ingress-canary-l5fhb\" (UID: \"3c28b757-a450-43a5-8c99-63a0e3824cc8\") " pod="openshift-ingress-canary/ingress-canary-l5fhb" Sep 11 14:15:23.463232 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.463194 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-bound-sa-token\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.497724 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.497525 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z5464"] Sep 11 14:15:23.501103 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.501081 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z5464" Sep 11 14:15:23.519003 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.518974 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 14:15:23.519129 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.519041 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xhc84\"" Sep 11 14:15:23.519129 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.519058 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 14:15:23.564324 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.564290 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls\") pod \"dns-default-z5464\" (UID: \"9e998861-da84-467d-a4e8-e959e24c336f\") " pod="openshift-dns/dns-default-z5464" Sep 11 14:15:23.564324 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.564328 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e998861-da84-467d-a4e8-e959e24c336f-tmp-dir\") pod \"dns-default-z5464\" (UID: \"9e998861-da84-467d-a4e8-e959e24c336f\") " pod="openshift-dns/dns-default-z5464" Sep 11 14:15:23.564873 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.564358 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-certificates\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.564873 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.564423 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb49m\" (UniqueName: \"kubernetes.io/projected/9e998861-da84-467d-a4e8-e959e24c336f-kube-api-access-wb49m\") pod \"dns-default-z5464\" (UID: \"9e998861-da84-467d-a4e8-e959e24c336f\") " pod="openshift-dns/dns-default-z5464" Sep 11 14:15:23.564873 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.564452 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c28b757-a450-43a5-8c99-63a0e3824cc8-cert\") pod \"ingress-canary-l5fhb\" (UID: \"3c28b757-a450-43a5-8c99-63a0e3824cc8\") " pod="openshift-ingress-canary/ingress-canary-l5fhb" Sep 11 14:15:23.564873 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.564484 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f32b4da3-04be-4cf9-94b9-03e8ab111b69-installation-pull-secrets\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.564873 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.564512 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f32b4da3-04be-4cf9-94b9-03e8ab111b69-image-registry-private-configuration\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.564873 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.564712 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e998861-da84-467d-a4e8-e959e24c336f-config-volume\") pod \"dns-default-z5464\" (UID: \"9e998861-da84-467d-a4e8-e959e24c336f\") " pod="openshift-dns/dns-default-z5464" Sep 11 14:15:23.564873 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.564777 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.564873 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.564807 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f32b4da3-04be-4cf9-94b9-03e8ab111b69-ca-trust-extracted\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.564873 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.564840 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ltmz\" (UniqueName: \"kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-kube-api-access-5ltmz\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.564873 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.564867 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f32b4da3-04be-4cf9-94b9-03e8ab111b69-trusted-ca\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.565326 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.564889 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96kbk\" (UniqueName: \"kubernetes.io/projected/3c28b757-a450-43a5-8c99-63a0e3824cc8-kube-api-access-96kbk\") pod \"ingress-canary-l5fhb\" (UID: \"3c28b757-a450-43a5-8c99-63a0e3824cc8\") " pod="openshift-ingress-canary/ingress-canary-l5fhb" Sep 11 14:15:23.565326 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.564929 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-bound-sa-token\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.565326 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.565078 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-certificates\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.565511 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.565363 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f32b4da3-04be-4cf9-94b9-03e8ab111b69-ca-trust-extracted\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.565511 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:23.565407 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 14:15:23.565511 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:23.565439 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74c56b68bf-b92lx: secret "image-registry-tls" not found Sep 11 14:15:23.565511 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:23.565489 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls podName:f32b4da3-04be-4cf9-94b9-03e8ab111b69 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:24.065471555 +0000 UTC m=+34.679577891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls") pod "image-registry-74c56b68bf-b92lx" (UID: "f32b4da3-04be-4cf9-94b9-03e8ab111b69") : secret "image-registry-tls" not found Sep 11 14:15:23.566115 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.566090 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f32b4da3-04be-4cf9-94b9-03e8ab111b69-trusted-ca\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.569111 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.569084 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f32b4da3-04be-4cf9-94b9-03e8ab111b69-image-registry-private-configuration\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.569236 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.569122 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f32b4da3-04be-4cf9-94b9-03e8ab111b69-installation-pull-secrets\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.569236 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.569213 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c28b757-a450-43a5-8c99-63a0e3824cc8-cert\") pod \"ingress-canary-l5fhb\" (UID: \"3c28b757-a450-43a5-8c99-63a0e3824cc8\") " pod="openshift-ingress-canary/ingress-canary-l5fhb" Sep 11 14:15:23.574312 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.574267 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z5464"] Sep 11 14:15:23.595645 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.595620 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-bound-sa-token\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.606392 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.606367 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96kbk\" (UniqueName: \"kubernetes.io/projected/3c28b757-a450-43a5-8c99-63a0e3824cc8-kube-api-access-96kbk\") pod \"ingress-canary-l5fhb\" (UID: \"3c28b757-a450-43a5-8c99-63a0e3824cc8\") " pod="openshift-ingress-canary/ingress-canary-l5fhb" Sep 11 14:15:23.609956 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.609930 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ltmz\" (UniqueName: \"kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-kube-api-access-5ltmz\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:23.666084 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.666047 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls\") pod \"dns-default-z5464\" (UID: \"9e998861-da84-467d-a4e8-e959e24c336f\") " pod="openshift-dns/dns-default-z5464" Sep 11 14:15:23.666252 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.666092 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e998861-da84-467d-a4e8-e959e24c336f-tmp-dir\") pod \"dns-default-z5464\" (UID: \"9e998861-da84-467d-a4e8-e959e24c336f\") " pod="openshift-dns/dns-default-z5464" Sep 11 14:15:23.666252 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:23.666204 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:23.666363 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.666246 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb49m\" (UniqueName: \"kubernetes.io/projected/9e998861-da84-467d-a4e8-e959e24c336f-kube-api-access-wb49m\") pod \"dns-default-z5464\" (UID: \"9e998861-da84-467d-a4e8-e959e24c336f\") " pod="openshift-dns/dns-default-z5464" Sep 11 14:15:23.666363 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:23.666268 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls podName:9e998861-da84-467d-a4e8-e959e24c336f nodeName:}" failed. No retries permitted until 2026-09-11 14:15:24.166246574 +0000 UTC m=+34.780352916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls") pod "dns-default-z5464" (UID: "9e998861-da84-467d-a4e8-e959e24c336f") : secret "dns-default-metrics-tls" not found Sep 11 14:15:23.666363 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.666314 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e998861-da84-467d-a4e8-e959e24c336f-config-volume\") pod \"dns-default-z5464\" (UID: \"9e998861-da84-467d-a4e8-e959e24c336f\") " pod="openshift-dns/dns-default-z5464" Sep 11 14:15:23.666528 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.666410 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e998861-da84-467d-a4e8-e959e24c336f-tmp-dir\") pod \"dns-default-z5464\" (UID: \"9e998861-da84-467d-a4e8-e959e24c336f\") " pod="openshift-dns/dns-default-z5464" Sep 11 14:15:23.666905 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.666889 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e998861-da84-467d-a4e8-e959e24c336f-config-volume\") pod \"dns-default-z5464\" (UID: \"9e998861-da84-467d-a4e8-e959e24c336f\") " pod="openshift-dns/dns-default-z5464" Sep 11 14:15:23.686687 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.686663 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb49m\" (UniqueName: \"kubernetes.io/projected/9e998861-da84-467d-a4e8-e959e24c336f-kube-api-access-wb49m\") pod \"dns-default-z5464\" (UID: \"9e998861-da84-467d-a4e8-e959e24c336f\") " pod="openshift-dns/dns-default-z5464" Sep 11 14:15:23.690466 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.690440 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l5fhb" Sep 11 14:15:23.839117 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.838985 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:23.969791 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:23.969753 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:24.068725 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:24.068684 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:24.068911 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:24.068850 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 14:15:24.068911 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:24.068868 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74c56b68bf-b92lx: secret "image-registry-tls" not found Sep 11 14:15:24.069017 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:24.068954 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls podName:f32b4da3-04be-4cf9-94b9-03e8ab111b69 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:25.068931751 +0000 UTC m=+35.683038097 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls") pod "image-registry-74c56b68bf-b92lx" (UID: "f32b4da3-04be-4cf9-94b9-03e8ab111b69") : secret "image-registry-tls" not found Sep 11 14:15:24.170204 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:24.170120 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls\") pod \"dns-default-z5464\" (UID: \"9e998861-da84-467d-a4e8-e959e24c336f\") " pod="openshift-dns/dns-default-z5464" Sep 11 14:15:24.170372 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:24.170278 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:24.170372 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:24.170342 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls podName:9e998861-da84-467d-a4e8-e959e24c336f nodeName:}" failed. No retries permitted until 2026-09-11 14:15:25.170326589 +0000 UTC m=+35.784432936 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls") pod "dns-default-z5464" (UID: "9e998861-da84-467d-a4e8-e959e24c336f") : secret "dns-default-metrics-tls" not found Sep 11 14:15:24.600370 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:24.600193 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l5fhb"] Sep 11 14:15:24.604604 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:15:24.604577 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c28b757_a450_43a5_8c99_63a0e3824cc8.slice/crio-ada0d2823a4f63dc6c3c09320e54883412e5c4714d4715c64e6684fd1338f7b0 WatchSource:0}: Error finding container ada0d2823a4f63dc6c3c09320e54883412e5c4714d4715c64e6684fd1338f7b0: Status 404 returned error can't find the container with id ada0d2823a4f63dc6c3c09320e54883412e5c4714d4715c64e6684fd1338f7b0 Sep 11 14:15:24.861925 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:24.861837 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:24.906238 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:24.906210 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:15:24.906381 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:24.906210 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:15:24.906452 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:24.906210 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:15:24.909892 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:24.909878 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cqjkr\"" Sep 11 14:15:24.910698 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:24.910680 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 14:15:24.910698 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:24.910695 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 14:15:24.911165 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:24.911149 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 14:15:24.911200 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:24.911171 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 14:15:24.911200 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:24.911191 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lrm6v\"" Sep 11 14:15:24.960647 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:24.960623 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:25.056211 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:25.056175 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:25.080681 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:25.080172 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:25.080681 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:25.080289 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 14:15:25.080681 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:25.080305 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74c56b68bf-b92lx: secret "image-registry-tls" not found Sep 11 14:15:25.080681 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:25.080365 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls podName:f32b4da3-04be-4cf9-94b9-03e8ab111b69 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:27.080346363 +0000 UTC m=+37.694452691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls") pod "image-registry-74c56b68bf-b92lx" (UID: "f32b4da3-04be-4cf9-94b9-03e8ab111b69") : secret "image-registry-tls" not found Sep 11 14:15:25.181721 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:25.181635 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls\") pod \"dns-default-z5464\" (UID: \"9e998861-da84-467d-a4e8-e959e24c336f\") " pod="openshift-dns/dns-default-z5464" Sep 11 14:15:25.181886 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:25.181810 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:25.181950 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:25.181887 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls podName:9e998861-da84-467d-a4e8-e959e24c336f nodeName:}" failed. No retries permitted until 2026-09-11 14:15:27.181866391 +0000 UTC m=+37.795972735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls") pod "dns-default-z5464" (UID: "9e998861-da84-467d-a4e8-e959e24c336f") : secret "dns-default-metrics-tls" not found Sep 11 14:15:25.230153 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:25.230117 2553 generic.go:356] "Generic (PLEG): container finished" podID="1cc0d3d9-09b0-48f9-99b7-2a11488eed7a" containerID="e605ca717d47f81c84cb4ba80f2b5ad82de30c24439c5be0e7e66dbb88092934" exitCode=0 Sep 11 14:15:25.230313 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:25.230206 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8jm26" event={"ID":"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a","Type":"ContainerDied","Data":"e605ca717d47f81c84cb4ba80f2b5ad82de30c24439c5be0e7e66dbb88092934"} Sep 11 14:15:25.231557 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:25.231535 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l5fhb" event={"ID":"3c28b757-a450-43a5-8c99-63a0e3824cc8","Type":"ContainerStarted","Data":"ada0d2823a4f63dc6c3c09320e54883412e5c4714d4715c64e6684fd1338f7b0"} Sep 11 14:15:25.880704 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:25.880642 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:25.985941 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:25.985899 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:26.084486 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:26.084450 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:26.237539 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:26.237448 2553 generic.go:356] "Generic (PLEG): container finished" podID="1cc0d3d9-09b0-48f9-99b7-2a11488eed7a" containerID="f90a7827b8b9bc72bf7f2087ae0d9ca0f2fe1e01ed8d74eaac3c149826e72221" exitCode=0 Sep 11 14:15:26.237680 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:26.237530 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8jm26" event={"ID":"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a","Type":"ContainerDied","Data":"f90a7827b8b9bc72bf7f2087ae0d9ca0f2fe1e01ed8d74eaac3c149826e72221"} Sep 11 14:15:26.670039 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:26.669949 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:27.096177 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:27.096128 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:27.096990 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:27.096646 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 14:15:27.096990 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:27.096672 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74c56b68bf-b92lx: secret "image-registry-tls" not found Sep 11 14:15:27.096990 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:27.096742 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls podName:f32b4da3-04be-4cf9-94b9-03e8ab111b69 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:31.096715563 +0000 UTC m=+41.710821900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls") pod "image-registry-74c56b68bf-b92lx" (UID: "f32b4da3-04be-4cf9-94b9-03e8ab111b69") : secret "image-registry-tls" not found Sep 11 14:15:27.196842 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:27.196768 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls\") pod \"dns-default-z5464\" (UID: \"9e998861-da84-467d-a4e8-e959e24c336f\") " pod="openshift-dns/dns-default-z5464" Sep 11 14:15:27.196979 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:27.196879 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:27.196979 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:27.196927 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls podName:9e998861-da84-467d-a4e8-e959e24c336f nodeName:}" failed. No retries permitted until 2026-09-11 14:15:31.196915479 +0000 UTC m=+41.811021806 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls") pod "dns-default-z5464" (UID: "9e998861-da84-467d-a4e8-e959e24c336f") : secret "dns-default-metrics-tls" not found Sep 11 14:15:27.243519 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:27.243484 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8jm26" event={"ID":"1cc0d3d9-09b0-48f9-99b7-2a11488eed7a","Type":"ContainerStarted","Data":"617616c1b9b255d0144eaf71cda9c1d307eeb233b397e409dff6670bee1e42c5"} Sep 11 14:15:27.244991 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:27.244963 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l5fhb" event={"ID":"3c28b757-a450-43a5-8c99-63a0e3824cc8","Type":"ContainerStarted","Data":"e0f3ee8f08978ec10775e23dee135274e796365fc12c128c3e3929fa92c894cd"} Sep 11 14:15:27.271062 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:27.271013 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8jm26" podStartSLOduration=7.782267719 podStartE2EDuration="37.270995289s" totalImagesPullingTime="29.48872757s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:27.270531842 +0000 UTC m=+37.884638186" watchObservedRunningTime="2026-09-11 14:15:27.270995289 +0000 UTC m=+37.885101627" Sep 11 14:15:27.290197 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:27.290151 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-l5fhb" podStartSLOduration=2.048528741 podStartE2EDuration="4.290137088s" totalImagesPullingTime="2.241608347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:27.289486194 +0000 UTC m=+37.903592539" watchObservedRunningTime="2026-09-11 14:15:27.290137088 +0000 UTC m=+37.904243489" Sep 11 14:15:28.722991 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:28.722951 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:29.117375 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:29.117297 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:29.186022 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:29.185990 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:30.893123 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:30.893084 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:31.124302 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:31.124267 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:31.124453 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:31.124430 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 14:15:31.124453 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:31.124447 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74c56b68bf-b92lx: secret "image-registry-tls" not found Sep 11 14:15:31.124524 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:31.124502 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls podName:f32b4da3-04be-4cf9-94b9-03e8ab111b69 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:39.12448461 +0000 UTC m=+49.738590931 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls") pod "image-registry-74c56b68bf-b92lx" (UID: "f32b4da3-04be-4cf9-94b9-03e8ab111b69") : secret "image-registry-tls" not found Sep 11 14:15:31.225070 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:31.224997 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls\") pod \"dns-default-z5464\" (UID: \"9e998861-da84-467d-a4e8-e959e24c336f\") " pod="openshift-dns/dns-default-z5464" Sep 11 14:15:31.225172 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:31.225137 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:31.225215 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:31.225204 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls podName:9e998861-da84-467d-a4e8-e959e24c336f nodeName:}" failed. No retries permitted until 2026-09-11 14:15:39.225188319 +0000 UTC m=+49.839294642 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls") pod "dns-default-z5464" (UID: "9e998861-da84-467d-a4e8-e959e24c336f") : secret "dns-default-metrics-tls" not found Sep 11 14:15:31.394763 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:31.394728 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:37.664103 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:37.664072 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:37.767843 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:37.767807 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:38.810915 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:38.810881 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:38.919363 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:38.919330 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:39.179604 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:39.179572 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:39.179766 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:39.179693 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 14:15:39.179766 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:39.179705 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74c56b68bf-b92lx: secret "image-registry-tls" not found Sep 11 14:15:39.179766 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:39.179758 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls podName:f32b4da3-04be-4cf9-94b9-03e8ab111b69 nodeName:}" failed. No retries permitted until 2026-09-11 14:15:55.179744072 +0000 UTC m=+65.793850394 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls") pod "image-registry-74c56b68bf-b92lx" (UID: "f32b4da3-04be-4cf9-94b9-03e8ab111b69") : secret "image-registry-tls" not found Sep 11 14:15:39.280545 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:39.280510 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls\") pod \"dns-default-z5464\" (UID: \"9e998861-da84-467d-a4e8-e959e24c336f\") " pod="openshift-dns/dns-default-z5464" Sep 11 14:15:39.280725 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:39.280665 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:39.280771 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:39.280736 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls podName:9e998861-da84-467d-a4e8-e959e24c336f nodeName:}" failed. No retries permitted until 2026-09-11 14:15:55.280719676 +0000 UTC m=+65.894826012 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls") pod "dns-default-z5464" (UID: "9e998861-da84-467d-a4e8-e959e24c336f") : secret "dns-default-metrics-tls" not found Sep 11 14:15:40.619060 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:40.619027 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:41.298166 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:41.298133 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:41.633381 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:41.633286 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:41.718951 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:41.718922 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:42.025348 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:42.025321 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:42.117225 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:42.117197 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:42.220290 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:42.220263 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:44.395830 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:44.395799 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:44.467228 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:44.467192 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:49.222811 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:49.222784 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mmqjh" Sep 11 14:15:49.401084 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:49.401048 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:49.463058 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:49.463026 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:54.585565 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:54.585528 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs\") pod \"network-metrics-daemon-s2sfs\" (UID: \"05960fb2-54e8-48b2-852b-caabc3c413ca\") " pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:15:54.588948 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:54.588928 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 14:15:54.596628 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:54.596610 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 11 14:15:54.596721 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:54.596664 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs podName:05960fb2-54e8-48b2-852b-caabc3c413ca nodeName:}" failed. No retries permitted until 2026-09-11 14:16:58.596648709 +0000 UTC m=+129.210755030 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs") pod "network-metrics-daemon-s2sfs" (UID: "05960fb2-54e8-48b2-852b-caabc3c413ca") : secret "metrics-daemon-secret" not found Sep 11 14:15:54.686379 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:54.686333 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret\") pod \"global-pull-secret-syncer-n7x6t\" (UID: \"bbb4cb01-8e27-4823-9410-d5b11334c3a9\") " pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:15:54.686379 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:54.686387 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdd8n\" (UniqueName: \"kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n\") pod \"network-check-target-jgdkq\" (UID: \"7a80c038-1fe1-4564-875f-e64fa2c5b2a6\") " pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:15:54.690088 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:54.690070 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 14:15:54.690353 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:54.690335 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 14:15:54.700404 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:54.700369 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bbb4cb01-8e27-4823-9410-d5b11334c3a9-original-pull-secret\") pod \"global-pull-secret-syncer-n7x6t\" (UID: \"bbb4cb01-8e27-4823-9410-d5b11334c3a9\") " pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:15:54.700581 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:54.700566 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 14:15:54.710508 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:54.710487 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdd8n\" (UniqueName: \"kubernetes.io/projected/7a80c038-1fe1-4564-875f-e64fa2c5b2a6-kube-api-access-fdd8n\") pod \"network-check-target-jgdkq\" (UID: \"7a80c038-1fe1-4564-875f-e64fa2c5b2a6\") " pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:15:54.919952 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:54.919922 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cqjkr\"" Sep 11 14:15:54.926665 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:54.926645 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:15:54.926749 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:54.926722 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n7x6t" Sep 11 14:15:55.190945 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:55.190871 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:15:55.191077 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:55.191019 2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 14:15:55.191077 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:55.191033 2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74c56b68bf-b92lx: secret "image-registry-tls" not found Sep 11 14:15:55.191144 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:55.191084 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls podName:f32b4da3-04be-4cf9-94b9-03e8ab111b69 nodeName:}" failed. No retries permitted until 2026-09-11 14:16:27.191069844 +0000 UTC m=+97.805176179 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls") pod "image-registry-74c56b68bf-b92lx" (UID: "f32b4da3-04be-4cf9-94b9-03e8ab111b69") : secret "image-registry-tls" not found Sep 11 14:15:55.211044 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:55.211015 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n7x6t"] Sep 11 14:15:55.214858 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:55.214836 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jgdkq"] Sep 11 14:15:55.214992 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:15:55.214895 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb4cb01_8e27_4823_9410_d5b11334c3a9.slice/crio-09c3187d7276b24075c157307396061354721942cb78f8b949aeae7cea034077 WatchSource:0}: Error finding container 09c3187d7276b24075c157307396061354721942cb78f8b949aeae7cea034077: Status 404 returned error can't find the container with id 09c3187d7276b24075c157307396061354721942cb78f8b949aeae7cea034077 Sep 11 14:15:55.216799 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:15:55.216781 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a80c038_1fe1_4564_875f_e64fa2c5b2a6.slice/crio-c31ba4b6d627235ba2480246a249170262ae4a4e6467d1c6d94dff5e2f29e1ff WatchSource:0}: Error finding container c31ba4b6d627235ba2480246a249170262ae4a4e6467d1c6d94dff5e2f29e1ff: Status 404 returned error can't find the container with id c31ba4b6d627235ba2480246a249170262ae4a4e6467d1c6d94dff5e2f29e1ff Sep 11 14:15:55.291553 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:55.291517 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls\") pod \"dns-default-z5464\" (UID: \"9e998861-da84-467d-a4e8-e959e24c336f\") " pod="openshift-dns/dns-default-z5464" Sep 11 14:15:55.291711 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:55.291662 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 14:15:55.291751 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:15:55.291721 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls podName:9e998861-da84-467d-a4e8-e959e24c336f nodeName:}" failed. No retries permitted until 2026-09-11 14:16:27.291705019 +0000 UTC m=+97.905811348 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls") pod "dns-default-z5464" (UID: "9e998861-da84-467d-a4e8-e959e24c336f") : secret "dns-default-metrics-tls" not found Sep 11 14:15:55.309357 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:55.309323 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n7x6t" event={"ID":"bbb4cb01-8e27-4823-9410-d5b11334c3a9","Type":"ContainerStarted","Data":"09c3187d7276b24075c157307396061354721942cb78f8b949aeae7cea034077"} Sep 11 14:15:55.310447 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:55.310406 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jgdkq" event={"ID":"7a80c038-1fe1-4564-875f-e64fa2c5b2a6","Type":"ContainerStarted","Data":"c31ba4b6d627235ba2480246a249170262ae4a4e6467d1c6d94dff5e2f29e1ff"} Sep 11 14:15:57.415218 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:15:57.415181 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:00.325222 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:00.325180 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n7x6t" event={"ID":"bbb4cb01-8e27-4823-9410-d5b11334c3a9","Type":"ContainerStarted","Data":"2a0254c9c94a10da9a364213960ce8a01f2e2e3d3efda44786af67590f1dcc2b"} Sep 11 14:16:00.326658 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:00.326634 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jgdkq" event={"ID":"7a80c038-1fe1-4564-875f-e64fa2c5b2a6","Type":"ContainerStarted","Data":"e4491ae1d27eeba138b8bcd5e787e900be07b0a58354947617cce3cef3da5ee5"} Sep 11 14:16:00.326789 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:00.326769 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:16:00.373238 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:00.373187 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-n7x6t" podStartSLOduration=66.014122628 podStartE2EDuration="1m10.373171996s" totalImagesPullingTime="4.359049368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:00.346511499 +0000 UTC m=+70.960617843" watchObservedRunningTime="2026-09-11 14:16:00.373171996 +0000 UTC m=+70.987278322" Sep 11 14:16:04.703821 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:04.703782 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:04.822918 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:04.822885 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:04.929721 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:04.929684 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:05.055624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:05.055526 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:05.131231 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:05.131200 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:05.227075 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:05.227041 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:05.348015 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:05.347911 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jgdkq" podStartSLOduration=70.994424405 podStartE2EDuration="1m15.347892642s" totalImagesPullingTime="4.353468237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:00.374751623 +0000 UTC m=+70.988857979" watchObservedRunningTime="2026-09-11 14:16:05.347892642 +0000 UTC m=+75.961998977" Sep 11 14:16:05.348443 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:05.348407 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dvlrt"] Sep 11 14:16:05.350767 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:05.350750 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dvlrt" Sep 11 14:16:05.362538 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:05.362517 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:16:05.362654 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:05.362518 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-k2w97\"" Sep 11 14:16:05.363690 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:05.363673 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-442bj\" (UniqueName: \"kubernetes.io/projected/79a83328-5a3b-4302-a708-7c4d7152102b-kube-api-access-442bj\") pod \"volume-data-source-validator-86695b7cc7-dvlrt\" (UID: \"79a83328-5a3b-4302-a708-7c4d7152102b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dvlrt" Sep 11 14:16:05.370646 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:05.370630 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 11 14:16:05.442660 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:05.442620 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dvlrt"] Sep 11 14:16:05.464149 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:05.464117 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-442bj\" (UniqueName: \"kubernetes.io/projected/79a83328-5a3b-4302-a708-7c4d7152102b-kube-api-access-442bj\") pod \"volume-data-source-validator-86695b7cc7-dvlrt\" (UID: \"79a83328-5a3b-4302-a708-7c4d7152102b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dvlrt" Sep 11 14:16:05.489273 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:05.489246 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-442bj\" (UniqueName: \"kubernetes.io/projected/79a83328-5a3b-4302-a708-7c4d7152102b-kube-api-access-442bj\") pod \"volume-data-source-validator-86695b7cc7-dvlrt\" (UID: \"79a83328-5a3b-4302-a708-7c4d7152102b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dvlrt" Sep 11 14:16:05.658836 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:05.658807 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dvlrt" Sep 11 14:16:05.981645 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:05.981569 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dvlrt"] Sep 11 14:16:05.984911 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:16:05.984879 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a83328_5a3b_4302_a708_7c4d7152102b.slice/crio-fe2c3432d4d2d5660bfda2e99093832df98ab31ac6dbedd93e5a44172bf87a44 WatchSource:0}: Error finding container fe2c3432d4d2d5660bfda2e99093832df98ab31ac6dbedd93e5a44172bf87a44: Status 404 returned error can't find the container with id fe2c3432d4d2d5660bfda2e99093832df98ab31ac6dbedd93e5a44172bf87a44 Sep 11 14:16:06.286590 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:06.286506 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:06.343055 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:06.343023 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dvlrt" event={"ID":"79a83328-5a3b-4302-a708-7c4d7152102b","Type":"ContainerStarted","Data":"fe2c3432d4d2d5660bfda2e99093832df98ab31ac6dbedd93e5a44172bf87a44"} Sep 11 14:16:06.375327 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:06.375293 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:07.346826 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:07.346792 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dvlrt" event={"ID":"79a83328-5a3b-4302-a708-7c4d7152102b","Type":"ContainerStarted","Data":"a3b96dc34da2130a152242818358677927bfdc5254fa678d4935b54d7f78481c"} Sep 11 14:16:07.370585 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:07.370536 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-dvlrt" podStartSLOduration=1.5140171919999998 podStartE2EDuration="2.370521198s" totalImagesPullingTime="856.504006ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:07.370321286 +0000 UTC m=+77.984427638" watchObservedRunningTime="2026-09-11 14:16:07.370521198 +0000 UTC m=+77.984627541" Sep 11 14:16:07.555279 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:07.555248 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:07.648875 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:07.648840 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:09.363967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:09.363932 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-59phf"] Sep 11 14:16:09.366222 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:09.366206 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-59phf" Sep 11 14:16:09.369619 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:09.369593 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ggg9r\"" Sep 11 14:16:09.369744 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:09.369707 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 11 14:16:09.370688 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:09.370669 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 11 14:16:09.386907 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:09.386884 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-59phf"] Sep 11 14:16:09.393327 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:09.393305 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c35b665e-20f7-4c00-a220-5b605f5f540d-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-59phf\" (UID: \"c35b665e-20f7-4c00-a220-5b605f5f540d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-59phf" Sep 11 14:16:09.393456 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:09.393346 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c35b665e-20f7-4c00-a220-5b605f5f540d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-59phf\" (UID: \"c35b665e-20f7-4c00-a220-5b605f5f540d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-59phf" Sep 11 14:16:09.494197 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:09.494162 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c35b665e-20f7-4c00-a220-5b605f5f540d-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-59phf\" (UID: \"c35b665e-20f7-4c00-a220-5b605f5f540d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-59phf" Sep 11 14:16:09.494391 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:09.494213 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c35b665e-20f7-4c00-a220-5b605f5f540d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-59phf\" (UID: \"c35b665e-20f7-4c00-a220-5b605f5f540d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-59phf" Sep 11 14:16:09.494391 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:09.494302 2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 14:16:09.494391 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:09.494358 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c35b665e-20f7-4c00-a220-5b605f5f540d-networking-console-plugin-cert podName:c35b665e-20f7-4c00-a220-5b605f5f540d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:09.994344067 +0000 UTC m=+80.608450388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c35b665e-20f7-4c00-a220-5b605f5f540d-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-59phf" (UID: "c35b665e-20f7-4c00-a220-5b605f5f540d") : secret "networking-console-plugin-cert" not found Sep 11 14:16:09.494827 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:09.494808 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c35b665e-20f7-4c00-a220-5b605f5f540d-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-59phf\" (UID: \"c35b665e-20f7-4c00-a220-5b605f5f540d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-59phf" Sep 11 14:16:09.997347 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:09.997314 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c35b665e-20f7-4c00-a220-5b605f5f540d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-59phf\" (UID: \"c35b665e-20f7-4c00-a220-5b605f5f540d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-59phf" Sep 11 14:16:09.997531 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:09.997475 2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 14:16:09.997570 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:09.997544 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c35b665e-20f7-4c00-a220-5b605f5f540d-networking-console-plugin-cert podName:c35b665e-20f7-4c00-a220-5b605f5f540d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:10.997521811 +0000 UTC m=+81.611628156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c35b665e-20f7-4c00-a220-5b605f5f540d-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-59phf" (UID: "c35b665e-20f7-4c00-a220-5b605f5f540d") : secret "networking-console-plugin-cert" not found Sep 11 14:16:11.005584 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.005545 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c35b665e-20f7-4c00-a220-5b605f5f540d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-59phf\" (UID: \"c35b665e-20f7-4c00-a220-5b605f5f540d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-59phf" Sep 11 14:16:11.006151 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:11.005731 2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 14:16:11.006151 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:11.005819 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c35b665e-20f7-4c00-a220-5b605f5f540d-networking-console-plugin-cert podName:c35b665e-20f7-4c00-a220-5b605f5f540d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:13.005796986 +0000 UTC m=+83.619903322 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c35b665e-20f7-4c00-a220-5b605f5f540d-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-59phf" (UID: "c35b665e-20f7-4c00-a220-5b605f5f540d") : secret "networking-console-plugin-cert" not found Sep 11 14:16:11.351477 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.351393 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:11.354510 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.354492 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-whsnc"] Sep 11 14:16:11.358655 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.358637 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:11.364573 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.364500 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 14:16:11.364653 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.364636 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 14:16:11.365493 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.365470 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lzjkv\"" Sep 11 14:16:11.365601 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.365529 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 14:16:11.365601 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.365530 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 14:16:11.387164 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.387134 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-whsnc"] Sep 11 14:16:11.408578 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.408551 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:11.408578 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.408581 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bxqk\" (UniqueName: \"kubernetes.io/projected/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-kube-api-access-9bxqk\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:11.408742 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.408601 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-crio-socket\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:11.408742 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.408662 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:11.408742 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.408719 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-data-volume\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:11.455608 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.455568 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:11.509831 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.509780 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-data-volume\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:11.510021 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.509856 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:11.510021 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.509875 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bxqk\" (UniqueName: \"kubernetes.io/projected/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-kube-api-access-9bxqk\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:11.510021 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.509894 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-crio-socket\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:11.510021 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.509928 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:11.510021 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:11.510009 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:11.510265 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.510023 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-crio-socket\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:11.510265 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:11.510015 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:16:11.510265 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:11.510077 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-insights-runtime-extractor-tls podName:af8a50bd-365d-4ed0-ae7a-7c45ef86224f nodeName:}" failed. No retries permitted until 2026-09-11 14:16:12.010062261 +0000 UTC m=+82.624168583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-whsnc" (UID: "af8a50bd-365d-4ed0-ae7a-7c45ef86224f") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:11.510265 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:11.510118 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-kube-rbac-proxy-cm podName:af8a50bd-365d-4ed0-ae7a-7c45ef86224f nodeName:}" failed. No retries permitted until 2026-09-11 14:16:12.010100356 +0000 UTC m=+82.624206704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-whsnc" (UID: "af8a50bd-365d-4ed0-ae7a-7c45ef86224f") : configmap "kube-rbac-proxy" not found Sep 11 14:16:11.510265 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.510171 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-data-volume\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:11.523622 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.523594 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bxqk\" (UniqueName: \"kubernetes.io/projected/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-kube-api-access-9bxqk\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:11.995945 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:11.995903 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:12.013989 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:12.013901 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:12.013989 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:12.013955 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:12.014333 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:12.014044 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:12.014333 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:12.014104 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-insights-runtime-extractor-tls podName:af8a50bd-365d-4ed0-ae7a-7c45ef86224f nodeName:}" failed. No retries permitted until 2026-09-11 14:16:13.014088685 +0000 UTC m=+83.628195020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-whsnc" (UID: "af8a50bd-365d-4ed0-ae7a-7c45ef86224f") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:12.014333 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:12.014044 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:16:12.014333 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:12.014186 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-kube-rbac-proxy-cm podName:af8a50bd-365d-4ed0-ae7a-7c45ef86224f nodeName:}" failed. No retries permitted until 2026-09-11 14:16:13.014170328 +0000 UTC m=+83.628276653 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-whsnc" (UID: "af8a50bd-365d-4ed0-ae7a-7c45ef86224f") : configmap "kube-rbac-proxy" not found Sep 11 14:16:12.093791 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:12.093752 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:12.424867 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:12.424841 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:12.496285 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:12.496259 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:12.930389 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:12.930358 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:13.021798 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:13.021766 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c35b665e-20f7-4c00-a220-5b605f5f540d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-59phf\" (UID: \"c35b665e-20f7-4c00-a220-5b605f5f540d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-59phf" Sep 11 14:16:13.021798 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:13.021800 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:13.022191 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:13.021877 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:13.022191 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:13.021919 2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 14:16:13.022191 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:13.021959 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:13.022191 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:13.021963 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:16:13.022191 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:13.021989 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c35b665e-20f7-4c00-a220-5b605f5f540d-networking-console-plugin-cert podName:c35b665e-20f7-4c00-a220-5b605f5f540d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:17.021969192 +0000 UTC m=+87.636075535 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c35b665e-20f7-4c00-a220-5b605f5f540d-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-59phf" (UID: "c35b665e-20f7-4c00-a220-5b605f5f540d") : secret "networking-console-plugin-cert" not found Sep 11 14:16:13.022191 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:13.022012 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-kube-rbac-proxy-cm podName:af8a50bd-365d-4ed0-ae7a-7c45ef86224f nodeName:}" failed. No retries permitted until 2026-09-11 14:16:15.021997509 +0000 UTC m=+85.636103853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-whsnc" (UID: "af8a50bd-365d-4ed0-ae7a-7c45ef86224f") : configmap "kube-rbac-proxy" not found Sep 11 14:16:13.022191 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:13.022029 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-insights-runtime-extractor-tls podName:af8a50bd-365d-4ed0-ae7a-7c45ef86224f nodeName:}" failed. No retries permitted until 2026-09-11 14:16:15.022021475 +0000 UTC m=+85.636127825 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-whsnc" (UID: "af8a50bd-365d-4ed0-ae7a-7c45ef86224f") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:13.039069 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:13.039044 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:13.158087 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:13.158059 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:13.249299 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:13.249212 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:13.480092 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:13.480064 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q25fm_7a24be49-54d5-46bf-b7ed-73b43b0bcb54/dns-node-resolver/0.log" Sep 11 14:16:14.278800 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:14.278773 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bn8wp_5159b394-744e-49c1-932d-a78367218c21/node-ca/0.log" Sep 11 14:16:14.763560 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:14.763533 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-chrjn"] Sep 11 14:16:14.766556 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:14.766540 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-chrjn" Sep 11 14:16:14.769664 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:14.769645 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 11 14:16:14.769977 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:14.769961 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 11 14:16:14.770888 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:14.770872 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 11 14:16:14.770971 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:14.770877 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 11 14:16:14.770971 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:14.770963 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fsksc\"" Sep 11 14:16:14.777285 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:14.777266 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-chrjn"] Sep 11 14:16:14.813338 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:14.813307 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:14.836988 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:14.836948 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/612a29bd-144f-46ac-9fe8-acf6f4252950-signing-key\") pod \"service-ca-568869fdc5-chrjn\" (UID: \"612a29bd-144f-46ac-9fe8-acf6f4252950\") " pod="openshift-service-ca/service-ca-568869fdc5-chrjn" Sep 11 14:16:14.837138 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:14.837061 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/612a29bd-144f-46ac-9fe8-acf6f4252950-signing-cabundle\") pod \"service-ca-568869fdc5-chrjn\" (UID: \"612a29bd-144f-46ac-9fe8-acf6f4252950\") " pod="openshift-service-ca/service-ca-568869fdc5-chrjn" Sep 11 14:16:14.837138 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:14.837109 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zscz\" (UniqueName: \"kubernetes.io/projected/612a29bd-144f-46ac-9fe8-acf6f4252950-kube-api-access-2zscz\") pod \"service-ca-568869fdc5-chrjn\" (UID: \"612a29bd-144f-46ac-9fe8-acf6f4252950\") " pod="openshift-service-ca/service-ca-568869fdc5-chrjn" Sep 11 14:16:14.916735 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:14.916704 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:14.938240 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:14.938198 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/612a29bd-144f-46ac-9fe8-acf6f4252950-signing-cabundle\") pod \"service-ca-568869fdc5-chrjn\" (UID: \"612a29bd-144f-46ac-9fe8-acf6f4252950\") " pod="openshift-service-ca/service-ca-568869fdc5-chrjn" Sep 11 14:16:14.938240 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:14.938242 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zscz\" (UniqueName: \"kubernetes.io/projected/612a29bd-144f-46ac-9fe8-acf6f4252950-kube-api-access-2zscz\") pod \"service-ca-568869fdc5-chrjn\" (UID: \"612a29bd-144f-46ac-9fe8-acf6f4252950\") " pod="openshift-service-ca/service-ca-568869fdc5-chrjn" Sep 11 14:16:14.938503 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:14.938400 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/612a29bd-144f-46ac-9fe8-acf6f4252950-signing-key\") pod \"service-ca-568869fdc5-chrjn\" (UID: \"612a29bd-144f-46ac-9fe8-acf6f4252950\") " pod="openshift-service-ca/service-ca-568869fdc5-chrjn" Sep 11 14:16:14.938934 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:14.938911 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/612a29bd-144f-46ac-9fe8-acf6f4252950-signing-cabundle\") pod \"service-ca-568869fdc5-chrjn\" (UID: \"612a29bd-144f-46ac-9fe8-acf6f4252950\") " pod="openshift-service-ca/service-ca-568869fdc5-chrjn" Sep 11 14:16:14.940747 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:14.940730 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/612a29bd-144f-46ac-9fe8-acf6f4252950-signing-key\") pod \"service-ca-568869fdc5-chrjn\" (UID: \"612a29bd-144f-46ac-9fe8-acf6f4252950\") " pod="openshift-service-ca/service-ca-568869fdc5-chrjn" Sep 11 14:16:14.952603 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:14.952585 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zscz\" (UniqueName: \"kubernetes.io/projected/612a29bd-144f-46ac-9fe8-acf6f4252950-kube-api-access-2zscz\") pod \"service-ca-568869fdc5-chrjn\" (UID: \"612a29bd-144f-46ac-9fe8-acf6f4252950\") " pod="openshift-service-ca/service-ca-568869fdc5-chrjn" Sep 11 14:16:15.039230 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:15.039151 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:15.039230 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:15.039205 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:15.039380 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:15.039285 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:16:15.039380 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:15.039312 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:15.039380 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:15.039335 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-kube-rbac-proxy-cm podName:af8a50bd-365d-4ed0-ae7a-7c45ef86224f nodeName:}" failed. No retries permitted until 2026-09-11 14:16:19.039321175 +0000 UTC m=+89.653427496 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-whsnc" (UID: "af8a50bd-365d-4ed0-ae7a-7c45ef86224f") : configmap "kube-rbac-proxy" not found Sep 11 14:16:15.039380 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:15.039368 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-insights-runtime-extractor-tls podName:af8a50bd-365d-4ed0-ae7a-7c45ef86224f nodeName:}" failed. No retries permitted until 2026-09-11 14:16:19.039355548 +0000 UTC m=+89.653461875 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-whsnc" (UID: "af8a50bd-365d-4ed0-ae7a-7c45ef86224f") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:15.075173 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:15.075139 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-chrjn" Sep 11 14:16:15.080993 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:15.080971 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l5fhb_3c28b757-a450-43a5-8c99-63a0e3824cc8/serve-healthcheck-canary/0.log" Sep 11 14:16:15.349406 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:15.349333 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-chrjn"] Sep 11 14:16:15.352219 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:16:15.352193 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612a29bd_144f_46ac_9fe8_acf6f4252950.slice/crio-7650c4f7729060317e8b498627c3623c0cdbf4c564b2910e3039e0a00ecddff6 WatchSource:0}: Error finding container 7650c4f7729060317e8b498627c3623c0cdbf4c564b2910e3039e0a00ecddff6: Status 404 returned error can't find the container with id 7650c4f7729060317e8b498627c3623c0cdbf4c564b2910e3039e0a00ecddff6 Sep 11 14:16:15.367912 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:15.367885 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-chrjn" event={"ID":"612a29bd-144f-46ac-9fe8-acf6f4252950","Type":"ContainerStarted","Data":"7650c4f7729060317e8b498627c3623c0cdbf4c564b2910e3039e0a00ecddff6"} Sep 11 14:16:16.227533 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:16.227493 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:17.057356 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:17.057329 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c35b665e-20f7-4c00-a220-5b605f5f540d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-59phf\" (UID: \"c35b665e-20f7-4c00-a220-5b605f5f540d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-59phf" Sep 11 14:16:17.057719 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:17.057511 2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 14:16:17.057719 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:17.057587 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c35b665e-20f7-4c00-a220-5b605f5f540d-networking-console-plugin-cert podName:c35b665e-20f7-4c00-a220-5b605f5f540d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:25.057564134 +0000 UTC m=+95.671670473 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c35b665e-20f7-4c00-a220-5b605f5f540d-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-59phf" (UID: "c35b665e-20f7-4c00-a220-5b605f5f540d") : secret "networking-console-plugin-cert" not found Sep 11 14:16:17.379195 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:17.379115 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-chrjn" event={"ID":"612a29bd-144f-46ac-9fe8-acf6f4252950","Type":"ContainerStarted","Data":"0eec357ddb4eeb475701d7a927f841b7919286b905d277756f23c4c8e7b72ca1"} Sep 11 14:16:18.854462 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:18.854428 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:19.050076 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:19.050020 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:19.074257 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:19.074219 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:19.074410 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:19.074310 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:19.074487 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:19.074469 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 14:16:19.074548 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:19.074533 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-kube-rbac-proxy-cm podName:af8a50bd-365d-4ed0-ae7a-7c45ef86224f nodeName:}" failed. No retries permitted until 2026-09-11 14:16:27.074514163 +0000 UTC m=+97.688620507 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-whsnc" (UID: "af8a50bd-365d-4ed0-ae7a-7c45ef86224f") : configmap "kube-rbac-proxy" not found Sep 11 14:16:19.074712 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:19.074646 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:19.074850 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:19.074841 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-insights-runtime-extractor-tls podName:af8a50bd-365d-4ed0-ae7a-7c45ef86224f nodeName:}" failed. No retries permitted until 2026-09-11 14:16:27.074813131 +0000 UTC m=+97.688919460 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-whsnc" (UID: "af8a50bd-365d-4ed0-ae7a-7c45ef86224f") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:19.149274 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:19.149237 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:19.252886 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:19.252850 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:19.376585 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:19.376507 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:19.513792 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:19.513704 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:19.617484 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:19.617449 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:19.757679 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:19.757647 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:19.949914 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:19.949864 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:20.940462 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:20.940428 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:21.348735 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:21.348647 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:21.540400 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:21.540363 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:21.825113 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:21.825072 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:22.148664 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:22.148631 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:22.552727 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:22.552639 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:22.623674 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:22.623640 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:23.002288 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:23.002247 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:23.152903 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:23.152870 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:23.392903 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:23.392872 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:23.477402 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:23.477371 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:23.716608 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:23.716520 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:23.804139 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:23.804109 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:24.596111 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:24.596075 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:25.121327 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:25.121297 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c35b665e-20f7-4c00-a220-5b605f5f540d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-59phf\" (UID: \"c35b665e-20f7-4c00-a220-5b605f5f540d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-59phf" Sep 11 14:16:25.121741 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:25.121462 2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 14:16:25.121741 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:25.121527 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c35b665e-20f7-4c00-a220-5b605f5f540d-networking-console-plugin-cert podName:c35b665e-20f7-4c00-a220-5b605f5f540d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:41.121511636 +0000 UTC m=+111.735617975 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c35b665e-20f7-4c00-a220-5b605f5f540d-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-59phf" (UID: "c35b665e-20f7-4c00-a220-5b605f5f540d") : secret "networking-console-plugin-cert" not found Sep 11 14:16:25.954368 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:25.954336 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:26.058011 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:26.057978 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:26.560829 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:26.560789 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:26.949849 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:26.949818 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:27.137341 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:27.137305 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:27.137553 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:27.137503 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:27.137626 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:27.137611 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 14:16:27.137681 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:27.137667 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-insights-runtime-extractor-tls podName:af8a50bd-365d-4ed0-ae7a-7c45ef86224f nodeName:}" failed. No retries permitted until 2026-09-11 14:16:43.137653493 +0000 UTC m=+113.751759844 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-whsnc" (UID: "af8a50bd-365d-4ed0-ae7a-7c45ef86224f") : secret "insights-runtime-extractor-tls" not found Sep 11 14:16:27.137968 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:27.137945 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:27.163837 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:27.163811 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:27.238712 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:27.238620 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:16:27.240996 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:27.240973 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls\") pod \"image-registry-74c56b68bf-b92lx\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:16:27.256685 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:27.256656 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w4lvw\"" Sep 11 14:16:27.264340 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:27.264321 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:16:27.340065 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:27.340028 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls\") pod \"dns-default-z5464\" (UID: \"9e998861-da84-467d-a4e8-e959e24c336f\") " pod="openshift-dns/dns-default-z5464" Sep 11 14:16:27.342194 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:27.342170 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e998861-da84-467d-a4e8-e959e24c336f-metrics-tls\") pod \"dns-default-z5464\" (UID: \"9e998861-da84-467d-a4e8-e959e24c336f\") " pod="openshift-dns/dns-default-z5464" Sep 11 14:16:27.351308 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:27.351283 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:27.413336 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:27.413307 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xhc84\"" Sep 11 14:16:27.421017 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:27.420989 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z5464" Sep 11 14:16:27.562514 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:27.562453 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-chrjn" podStartSLOduration=11.913959104 podStartE2EDuration="13.56243153s" totalImagesPullingTime="1.648472426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:17.412041158 +0000 UTC m=+88.026147496" watchObservedRunningTime="2026-09-11 14:16:27.56243153 +0000 UTC m=+98.176537868" Sep 11 14:16:27.564037 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:27.564008 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74c56b68bf-b92lx"] Sep 11 14:16:27.567169 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:16:27.567138 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32b4da3_04be_4cf9_94b9_03e8ab111b69.slice/crio-9f31de9fe2e6a9b521228fb5f9b22e3a09631ab3a7b3e58b2058c26076dd0a08 WatchSource:0}: Error finding container 9f31de9fe2e6a9b521228fb5f9b22e3a09631ab3a7b3e58b2058c26076dd0a08: Status 404 returned error can't find the container with id 9f31de9fe2e6a9b521228fb5f9b22e3a09631ab3a7b3e58b2058c26076dd0a08 Sep 11 14:16:27.598276 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:27.598238 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:27.717681 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:27.717645 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z5464"] Sep 11 14:16:27.721376 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:16:27.721350 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e998861_da84_467d_a4e8_e959e24c336f.slice/crio-39e09bdafee490449ae07a82ba96c3e09449c7843a7da464bdd2adaa7de78b00 WatchSource:0}: Error finding container 39e09bdafee490449ae07a82ba96c3e09449c7843a7da464bdd2adaa7de78b00: Status 404 returned error can't find the container with id 39e09bdafee490449ae07a82ba96c3e09449c7843a7da464bdd2adaa7de78b00 Sep 11 14:16:28.411844 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:28.411806 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z5464" event={"ID":"9e998861-da84-467d-a4e8-e959e24c336f","Type":"ContainerStarted","Data":"39e09bdafee490449ae07a82ba96c3e09449c7843a7da464bdd2adaa7de78b00"} Sep 11 14:16:28.413536 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:28.413508 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" event={"ID":"f32b4da3-04be-4cf9-94b9-03e8ab111b69","Type":"ContainerStarted","Data":"76d495a01055ba3bc465eb7927128defd89d37b1641137b525dc10fb6abcae50"} Sep 11 14:16:28.413684 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:28.413554 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" event={"ID":"f32b4da3-04be-4cf9-94b9-03e8ab111b69","Type":"ContainerStarted","Data":"9f31de9fe2e6a9b521228fb5f9b22e3a09631ab3a7b3e58b2058c26076dd0a08"} Sep 11 14:16:28.413684 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:28.413595 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:16:28.439981 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:28.439929 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" podStartSLOduration=98.439912506 podStartE2EDuration="1m38.439912506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:28.439297319 +0000 UTC m=+99.053403662" watchObservedRunningTime="2026-09-11 14:16:28.439912506 +0000 UTC m=+99.054018850" Sep 11 14:16:28.555431 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:28.555346 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:28.944500 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:28.944449 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:29.120329 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:29.120294 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:29.218328 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:29.218236 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:29.381141 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:29.381112 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:29.417775 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:29.417744 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z5464" event={"ID":"9e998861-da84-467d-a4e8-e959e24c336f","Type":"ContainerStarted","Data":"1ae8de931d87f907d0e3bba55f30ded2e23a42b7fbadd3f0dd926d5173bfad4c"} Sep 11 14:16:29.543064 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:29.542962 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:30.422214 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:30.422177 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z5464" event={"ID":"9e998861-da84-467d-a4e8-e959e24c336f","Type":"ContainerStarted","Data":"574053a5e42a1a3d588928366a14613634ef77eaa1dad0a26f86007dcbfe3411"} Sep 11 14:16:30.422700 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:30.422292 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z5464" Sep 11 14:16:31.321824 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:31.321791 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:31.331289 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:31.331268 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jgdkq" Sep 11 14:16:31.355348 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:31.355306 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-z5464" podStartSLOduration=66.816436232 podStartE2EDuration="1m8.355293032s" totalImagesPullingTime="1.5388568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:30.448660904 +0000 UTC m=+101.062767249" watchObservedRunningTime="2026-09-11 14:16:31.355293032 +0000 UTC m=+101.969399358" Sep 11 14:16:31.420796 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:31.420763 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:31.647427 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:31.647384 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:31.759335 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:31.759294 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:31.849752 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:31.849721 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:32.008109 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:32.008013 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:32.636619 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:32.636581 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:32.829486 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:32.829447 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:33.210478 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:33.210437 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:33.400532 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:33.400499 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:33.497952 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:33.497860 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:34.445435 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.445393 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:34.487184 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.487157 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cf49b5b8c-tb8vv"] Sep 11 14:16:34.490061 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.490038 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cf49b5b8c-tb8vv" Sep 11 14:16:34.491311 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.491290 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fd8df9b46-q4l8f"] Sep 11 14:16:34.492937 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.492923 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fd8df9b46-q4l8f" Sep 11 14:16:34.493770 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.493750 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 11 14:16:34.493901 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.493885 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 11 14:16:34.495196 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.495174 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 11 14:16:34.495288 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.495235 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 11 14:16:34.495523 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.495504 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-ztr6z\"" Sep 11 14:16:34.495615 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.495573 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 11 14:16:34.532394 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.532366 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cf49b5b8c-tb8vv"] Sep 11 14:16:34.533256 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.533234 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fd8df9b46-q4l8f"] Sep 11 14:16:34.597514 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.597479 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/acc06fe1-cada-4013-9298-2bda5659e4f0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-fd8df9b46-q4l8f\" (UID: \"acc06fe1-cada-4013-9298-2bda5659e4f0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fd8df9b46-q4l8f" Sep 11 14:16:34.597671 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.597521 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz4rn\" (UniqueName: \"kubernetes.io/projected/16e30b5a-5e84-43c6-9fb4-52ce0dca9cdf-kube-api-access-hz4rn\") pod \"klusterlet-addon-workmgr-6cf49b5b8c-tb8vv\" (UID: \"16e30b5a-5e84-43c6-9fb4-52ce0dca9cdf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cf49b5b8c-tb8vv" Sep 11 14:16:34.597671 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.597607 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnb6p\" (UniqueName: \"kubernetes.io/projected/acc06fe1-cada-4013-9298-2bda5659e4f0-kube-api-access-fnb6p\") pod \"managed-serviceaccount-addon-agent-fd8df9b46-q4l8f\" (UID: \"acc06fe1-cada-4013-9298-2bda5659e4f0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fd8df9b46-q4l8f" Sep 11 14:16:34.597671 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.597643 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16e30b5a-5e84-43c6-9fb4-52ce0dca9cdf-tmp\") pod \"klusterlet-addon-workmgr-6cf49b5b8c-tb8vv\" (UID: \"16e30b5a-5e84-43c6-9fb4-52ce0dca9cdf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cf49b5b8c-tb8vv" Sep 11 14:16:34.597771 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.597678 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/16e30b5a-5e84-43c6-9fb4-52ce0dca9cdf-klusterlet-config\") pod \"klusterlet-addon-workmgr-6cf49b5b8c-tb8vv\" (UID: \"16e30b5a-5e84-43c6-9fb4-52ce0dca9cdf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cf49b5b8c-tb8vv" Sep 11 14:16:34.692577 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.692545 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:34.698591 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.698507 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnb6p\" (UniqueName: \"kubernetes.io/projected/acc06fe1-cada-4013-9298-2bda5659e4f0-kube-api-access-fnb6p\") pod \"managed-serviceaccount-addon-agent-fd8df9b46-q4l8f\" (UID: \"acc06fe1-cada-4013-9298-2bda5659e4f0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fd8df9b46-q4l8f" Sep 11 14:16:34.698591 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.698551 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16e30b5a-5e84-43c6-9fb4-52ce0dca9cdf-tmp\") pod \"klusterlet-addon-workmgr-6cf49b5b8c-tb8vv\" (UID: \"16e30b5a-5e84-43c6-9fb4-52ce0dca9cdf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cf49b5b8c-tb8vv" Sep 11 14:16:34.698591 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.698587 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/16e30b5a-5e84-43c6-9fb4-52ce0dca9cdf-klusterlet-config\") pod \"klusterlet-addon-workmgr-6cf49b5b8c-tb8vv\" (UID: \"16e30b5a-5e84-43c6-9fb4-52ce0dca9cdf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cf49b5b8c-tb8vv" Sep 11 14:16:34.698846 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.698638 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/acc06fe1-cada-4013-9298-2bda5659e4f0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-fd8df9b46-q4l8f\" (UID: \"acc06fe1-cada-4013-9298-2bda5659e4f0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fd8df9b46-q4l8f" Sep 11 14:16:34.698846 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.698665 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz4rn\" (UniqueName: \"kubernetes.io/projected/16e30b5a-5e84-43c6-9fb4-52ce0dca9cdf-kube-api-access-hz4rn\") pod \"klusterlet-addon-workmgr-6cf49b5b8c-tb8vv\" (UID: \"16e30b5a-5e84-43c6-9fb4-52ce0dca9cdf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cf49b5b8c-tb8vv" Sep 11 14:16:34.699010 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.698992 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16e30b5a-5e84-43c6-9fb4-52ce0dca9cdf-tmp\") pod \"klusterlet-addon-workmgr-6cf49b5b8c-tb8vv\" (UID: \"16e30b5a-5e84-43c6-9fb4-52ce0dca9cdf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cf49b5b8c-tb8vv" Sep 11 14:16:34.701008 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.700989 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/16e30b5a-5e84-43c6-9fb4-52ce0dca9cdf-klusterlet-config\") pod \"klusterlet-addon-workmgr-6cf49b5b8c-tb8vv\" (UID: \"16e30b5a-5e84-43c6-9fb4-52ce0dca9cdf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cf49b5b8c-tb8vv" Sep 11 14:16:34.701109 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.701094 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/acc06fe1-cada-4013-9298-2bda5659e4f0-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-fd8df9b46-q4l8f\" (UID: \"acc06fe1-cada-4013-9298-2bda5659e4f0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fd8df9b46-q4l8f" Sep 11 14:16:34.722303 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.722281 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnb6p\" (UniqueName: \"kubernetes.io/projected/acc06fe1-cada-4013-9298-2bda5659e4f0-kube-api-access-fnb6p\") pod \"managed-serviceaccount-addon-agent-fd8df9b46-q4l8f\" (UID: \"acc06fe1-cada-4013-9298-2bda5659e4f0\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fd8df9b46-q4l8f" Sep 11 14:16:34.729029 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.729009 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz4rn\" (UniqueName: \"kubernetes.io/projected/16e30b5a-5e84-43c6-9fb4-52ce0dca9cdf-kube-api-access-hz4rn\") pod \"klusterlet-addon-workmgr-6cf49b5b8c-tb8vv\" (UID: \"16e30b5a-5e84-43c6-9fb4-52ce0dca9cdf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cf49b5b8c-tb8vv" Sep 11 14:16:34.800067 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.800024 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cf49b5b8c-tb8vv" Sep 11 14:16:34.805523 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.805493 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:34.814831 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:34.814811 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fd8df9b46-q4l8f" Sep 11 14:16:35.089488 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:35.089400 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cf49b5b8c-tb8vv"] Sep 11 14:16:35.092832 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:16:35.092806 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e30b5a_5e84_43c6_9fb4_52ce0dca9cdf.slice/crio-5dfc1a3d12ab0a020a7e792366d7f8ef04ad66fa38947d868d8c1a4678f6f44c WatchSource:0}: Error finding container 5dfc1a3d12ab0a020a7e792366d7f8ef04ad66fa38947d868d8c1a4678f6f44c: Status 404 returned error can't find the container with id 5dfc1a3d12ab0a020a7e792366d7f8ef04ad66fa38947d868d8c1a4678f6f44c Sep 11 14:16:35.107960 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:35.107828 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fd8df9b46-q4l8f"] Sep 11 14:16:35.110170 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:16:35.110147 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podacc06fe1_cada_4013_9298_2bda5659e4f0.slice/crio-ee46907e8e81f130ca0209cea7f2cb7f09918cf6cabb96323662a3ab44ed6e15 WatchSource:0}: Error finding container ee46907e8e81f130ca0209cea7f2cb7f09918cf6cabb96323662a3ab44ed6e15: Status 404 returned error can't find the container with id ee46907e8e81f130ca0209cea7f2cb7f09918cf6cabb96323662a3ab44ed6e15 Sep 11 14:16:35.441404 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:35.441369 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fd8df9b46-q4l8f" event={"ID":"acc06fe1-cada-4013-9298-2bda5659e4f0","Type":"ContainerStarted","Data":"ee46907e8e81f130ca0209cea7f2cb7f09918cf6cabb96323662a3ab44ed6e15"} Sep 11 14:16:35.442371 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:35.442352 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cf49b5b8c-tb8vv" event={"ID":"16e30b5a-5e84-43c6-9fb4-52ce0dca9cdf","Type":"ContainerStarted","Data":"5dfc1a3d12ab0a020a7e792366d7f8ef04ad66fa38947d868d8c1a4678f6f44c"} Sep 11 14:16:37.425553 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:37.425516 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:39.434606 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:39.434571 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:39.520183 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:39.520150 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:40.345575 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:40.345543 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:40.428352 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:40.428322 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z5464" Sep 11 14:16:40.440676 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:40.440650 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:40.464823 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:40.464795 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fd8df9b46-q4l8f" event={"ID":"acc06fe1-cada-4013-9298-2bda5659e4f0","Type":"ContainerStarted","Data":"bc96bd0300a3e52901161d66d8fe576ea8466ae4c7a28628b910bd0615f2e569"} Sep 11 14:16:40.466729 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:40.466705 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cf49b5b8c-tb8vv" event={"ID":"16e30b5a-5e84-43c6-9fb4-52ce0dca9cdf","Type":"ContainerStarted","Data":"df43c8004fb6e30fb8c7c4907354302809a261d9eaf2fd0199a0a551063af4eb"} Sep 11 14:16:40.466863 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:40.466821 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cf49b5b8c-tb8vv" Sep 11 14:16:40.468515 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:40.468491 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cf49b5b8c-tb8vv" Sep 11 14:16:40.496805 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:40.496763 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-fd8df9b46-q4l8f" podStartSLOduration=2.087654783 podStartE2EDuration="6.496748589s" totalImagesPullingTime="4.409093806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:40.49602896 +0000 UTC m=+111.110135286" watchObservedRunningTime="2026-09-11 14:16:40.496748589 +0000 UTC m=+111.110854932" Sep 11 14:16:40.525654 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:40.525611 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6cf49b5b8c-tb8vv" podStartSLOduration=2.038603459 podStartE2EDuration="6.525596461s" totalImagesPullingTime="4.486993002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:40.525082745 +0000 UTC m=+111.139189102" watchObservedRunningTime="2026-09-11 14:16:40.525596461 +0000 UTC m=+111.139702853" Sep 11 14:16:40.923314 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:40.923258 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:41.031471 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:41.031439 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:41.151020 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:41.150984 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c35b665e-20f7-4c00-a220-5b605f5f540d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-59phf\" (UID: \"c35b665e-20f7-4c00-a220-5b605f5f540d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-59phf" Sep 11 14:16:41.153318 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:41.153292 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c35b665e-20f7-4c00-a220-5b605f5f540d-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-59phf\" (UID: \"c35b665e-20f7-4c00-a220-5b605f5f540d\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-59phf" Sep 11 14:16:41.165280 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:41.165215 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:41.175136 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:41.175071 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-59phf" Sep 11 14:16:41.227868 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:41.227825 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pd78h"] Sep 11 14:16:41.231653 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:41.231630 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pd78h" Sep 11 14:16:41.235537 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:41.235509 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-cfqsb\"" Sep 11 14:16:41.237503 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:41.237293 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 11 14:16:41.247602 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:41.247579 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pd78h"] Sep 11 14:16:41.287311 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:41.287286 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:41.353038 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:41.353004 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c79d8dda-9f21-42df-a4fa-e350d0d381bb-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-pd78h\" (UID: \"c79d8dda-9f21-42df-a4fa-e350d0d381bb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pd78h" Sep 11 14:16:41.453674 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:41.453582 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c79d8dda-9f21-42df-a4fa-e350d0d381bb-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-pd78h\" (UID: \"c79d8dda-9f21-42df-a4fa-e350d0d381bb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pd78h" Sep 11 14:16:41.454813 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:41.454788 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-59phf"] Sep 11 14:16:41.456185 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:41.456168 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c79d8dda-9f21-42df-a4fa-e350d0d381bb-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-pd78h\" (UID: \"c79d8dda-9f21-42df-a4fa-e350d0d381bb\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pd78h" Sep 11 14:16:41.458124 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:16:41.458097 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35b665e_20f7_4c00_a220_5b605f5f540d.slice/crio-81faeb303d637bddb237a51187f285e5e9a00eaae19e63176a6974fafa9b7802 WatchSource:0}: Error finding container 81faeb303d637bddb237a51187f285e5e9a00eaae19e63176a6974fafa9b7802: Status 404 returned error can't find the container with id 81faeb303d637bddb237a51187f285e5e9a00eaae19e63176a6974fafa9b7802 Sep 11 14:16:41.470797 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:41.470768 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-59phf" event={"ID":"c35b665e-20f7-4c00-a220-5b605f5f540d","Type":"ContainerStarted","Data":"81faeb303d637bddb237a51187f285e5e9a00eaae19e63176a6974fafa9b7802"} Sep 11 14:16:41.543118 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:41.543083 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pd78h" Sep 11 14:16:41.820888 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:41.820861 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pd78h"] Sep 11 14:16:41.823410 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:16:41.823376 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79d8dda_9f21_42df_a4fa_e350d0d381bb.slice/crio-87a485ec4535f74f630d98ec72157652c055f1a22ce436ed9fc50ef85c0b429d WatchSource:0}: Error finding container 87a485ec4535f74f630d98ec72157652c055f1a22ce436ed9fc50ef85c0b429d: Status 404 returned error can't find the container with id 87a485ec4535f74f630d98ec72157652c055f1a22ce436ed9fc50ef85c0b429d Sep 11 14:16:41.869872 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:41.869838 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:42.476900 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:42.476769 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-59phf" event={"ID":"c35b665e-20f7-4c00-a220-5b605f5f540d","Type":"ContainerStarted","Data":"2b5c53a22c30c09f244e7d6dffa66a09b84ecaf34b292aa81ed678bae5538590"} Sep 11 14:16:42.478381 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:42.478352 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pd78h" event={"ID":"c79d8dda-9f21-42df-a4fa-e350d0d381bb","Type":"ContainerStarted","Data":"87a485ec4535f74f630d98ec72157652c055f1a22ce436ed9fc50ef85c0b429d"} Sep 11 14:16:42.502646 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:42.502558 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-59phf" podStartSLOduration=32.587981311 podStartE2EDuration="33.502538252s" totalImagesPullingTime="914.556941ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:42.501692546 +0000 UTC m=+113.115798892" watchObservedRunningTime="2026-09-11 14:16:42.502538252 +0000 UTC m=+113.116644596" Sep 11 14:16:42.656070 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:42.656032 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:42.752468 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:42.752436 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:43.166906 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:43.166868 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:43.169164 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:43.169144 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/af8a50bd-365d-4ed0-ae7a-7c45ef86224f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-whsnc\" (UID: \"af8a50bd-365d-4ed0-ae7a-7c45ef86224f\") " pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:43.466837 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:43.466797 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-whsnc" Sep 11 14:16:43.484594 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:43.484565 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pd78h" event={"ID":"c79d8dda-9f21-42df-a4fa-e350d0d381bb","Type":"ContainerStarted","Data":"271751f6251a82894ede4145dd6c7c72b1eacec533aac8a4cfd6ceae3781f376"} Sep 11 14:16:43.484954 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:43.484830 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pd78h" Sep 11 14:16:43.489680 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:43.489657 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pd78h" Sep 11 14:16:43.521751 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:43.521694 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-pd78h" podStartSLOduration=1.382441266 podStartE2EDuration="2.521671327s" totalImagesPullingTime="1.139230061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:43.521034155 +0000 UTC m=+114.135140522" watchObservedRunningTime="2026-09-11 14:16:43.521671327 +0000 UTC m=+114.135777672" Sep 11 14:16:43.759782 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:43.759696 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-whsnc"] Sep 11 14:16:43.762844 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:16:43.762817 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf8a50bd_365d_4ed0_ae7a_7c45ef86224f.slice/crio-ad12c8fd496c5bca9f64b81ec21d661b30402d2681f6b4a5d9b51f5a8d205268 WatchSource:0}: Error finding container ad12c8fd496c5bca9f64b81ec21d661b30402d2681f6b4a5d9b51f5a8d205268: Status 404 returned error can't find the container with id ad12c8fd496c5bca9f64b81ec21d661b30402d2681f6b4a5d9b51f5a8d205268 Sep 11 14:16:44.360893 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:44.360852 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:44.488524 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:44.488491 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-whsnc" event={"ID":"af8a50bd-365d-4ed0-ae7a-7c45ef86224f","Type":"ContainerStarted","Data":"44e7382be7e35c61e3cfc3f035bdfb7a42da822b32bddd0896cc63ef87917d76"} Sep 11 14:16:44.488786 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:44.488554 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-whsnc" event={"ID":"af8a50bd-365d-4ed0-ae7a-7c45ef86224f","Type":"ContainerStarted","Data":"ad12c8fd496c5bca9f64b81ec21d661b30402d2681f6b4a5d9b51f5a8d205268"} Sep 11 14:16:44.551315 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:44.551262 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:44.696888 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:44.696845 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:45.493778 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:45.493736 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-whsnc" event={"ID":"af8a50bd-365d-4ed0-ae7a-7c45ef86224f","Type":"ContainerStarted","Data":"cbd9e99e78565fc03dc6dfda272d2b0659ac84c2550fde6df347eebf02b07dd0"} Sep 11 14:16:46.480265 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:46.480234 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:46.498642 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:46.498612 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-whsnc" event={"ID":"af8a50bd-365d-4ed0-ae7a-7c45ef86224f","Type":"ContainerStarted","Data":"bcca8e0c8d5a2885ac1b3544e835a39edf754ce81ac5cbe1bd2dc29aa836de93"} Sep 11 14:16:46.559022 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:46.558973 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-whsnc" podStartSLOduration=33.449608321 podStartE2EDuration="35.558958072s" totalImagesPullingTime="2.109349751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:16:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:16:46.556099812 +0000 UTC m=+117.170206156" watchObservedRunningTime="2026-09-11 14:16:46.558958072 +0000 UTC m=+117.173064419" Sep 11 14:16:47.269020 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:47.268989 2553 patch_prober.go:28] interesting pod/image-registry-74c56b68bf-b92lx container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 11 14:16:47.269202 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:47.269046 2553 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" podUID="f32b4da3-04be-4cf9-94b9-03e8ab111b69" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:16:47.894976 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:47.894943 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:48.014612 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:48.014576 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:49.424395 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:49.424261 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:16:49.746451 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:49.746345 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:49.869993 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:49.869959 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:50.053328 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:50.053237 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:50.147824 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:50.147791 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:52.074005 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:52.073971 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:53.215662 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:53.215629 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:53.318993 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:53.318952 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:54.021750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:54.021719 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:54.121018 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:54.120980 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:54.678073 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:54.678044 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:55.850743 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:55.850705 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:55.960704 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:55.960674 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:56.686996 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:56.686960 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:57.061006 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.060924 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:57.080112 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.080080 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74c56b68bf-b92lx"] Sep 11 14:16:57.213925 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.213883 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:57.389339 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.389308 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:57.535850 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.535822 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp"] Sep 11 14:16:57.540267 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.540249 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp" Sep 11 14:16:57.546060 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.546036 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 11 14:16:57.546212 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.546192 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 14:16:57.547056 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.547034 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 14:16:57.547171 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.547034 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 11 14:16:57.547278 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.547255 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4tp5b\"" Sep 11 14:16:57.547718 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.547703 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 14:16:57.631823 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.631796 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kgzrk"] Sep 11 14:16:57.637970 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.637937 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp"] Sep 11 14:16:57.638642 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.638619 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.657806 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:57.657777 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-kube-rbac-proxy-config\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" Sep 11 14:16:57.661790 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:57.661771 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-tls\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" Sep 11 14:16:57.676608 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:57.676575 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-dockercfg-tsp77\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tsp77\"" type="*v1.Secret" Sep 11 14:16:57.676713 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:57.676657 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"node-exporter-accelerators-collector-config\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" Sep 11 14:16:57.680125 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.680106 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95c23d32-5f8b-4fbd-8e68-dd65447d578d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-cx7cp\" (UID: \"95c23d32-5f8b-4fbd-8e68-dd65447d578d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp" Sep 11 14:16:57.680194 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.680151 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95c23d32-5f8b-4fbd-8e68-dd65447d578d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-cx7cp\" (UID: \"95c23d32-5f8b-4fbd-8e68-dd65447d578d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp" Sep 11 14:16:57.680194 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.680180 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cclbb\" (UniqueName: \"kubernetes.io/projected/95c23d32-5f8b-4fbd-8e68-dd65447d578d-kube-api-access-cclbb\") pod \"openshift-state-metrics-67dcf8c979-cx7cp\" (UID: \"95c23d32-5f8b-4fbd-8e68-dd65447d578d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp" Sep 11 14:16:57.680295 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.680229 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95c23d32-5f8b-4fbd-8e68-dd65447d578d-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-cx7cp\" (UID: \"95c23d32-5f8b-4fbd-8e68-dd65447d578d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp" Sep 11 14:16:57.713700 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.713667 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:57.781160 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.781125 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/11604680-a734-4e85-81e3-8dc215a6fd4a-root\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.781160 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.781164 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95c23d32-5f8b-4fbd-8e68-dd65447d578d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-cx7cp\" (UID: \"95c23d32-5f8b-4fbd-8e68-dd65447d578d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp" Sep 11 14:16:57.781389 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.781208 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95c23d32-5f8b-4fbd-8e68-dd65447d578d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-cx7cp\" (UID: \"95c23d32-5f8b-4fbd-8e68-dd65447d578d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp" Sep 11 14:16:57.781389 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.781227 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11604680-a734-4e85-81e3-8dc215a6fd4a-sys\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.781389 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.781242 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-tls\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.781389 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.781274 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11604680-a734-4e85-81e3-8dc215a6fd4a-metrics-client-ca\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.781389 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.781304 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cclbb\" (UniqueName: \"kubernetes.io/projected/95c23d32-5f8b-4fbd-8e68-dd65447d578d-kube-api-access-cclbb\") pod \"openshift-state-metrics-67dcf8c979-cx7cp\" (UID: \"95c23d32-5f8b-4fbd-8e68-dd65447d578d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp" Sep 11 14:16:57.781389 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:57.781314 2553 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 11 14:16:57.781389 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.781347 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95c23d32-5f8b-4fbd-8e68-dd65447d578d-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-cx7cp\" (UID: \"95c23d32-5f8b-4fbd-8e68-dd65447d578d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp" Sep 11 14:16:57.781778 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:57.781398 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95c23d32-5f8b-4fbd-8e68-dd65447d578d-openshift-state-metrics-tls podName:95c23d32-5f8b-4fbd-8e68-dd65447d578d nodeName:}" failed. No retries permitted until 2026-09-11 14:16:58.281377125 +0000 UTC m=+128.895483464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/95c23d32-5f8b-4fbd-8e68-dd65447d578d-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-cx7cp" (UID: "95c23d32-5f8b-4fbd-8e68-dd65447d578d") : secret "openshift-state-metrics-tls" not found Sep 11 14:16:57.781778 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.781454 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.781778 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.781489 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.781778 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.781540 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-wtmp\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.781778 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.781584 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-textfile\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.781778 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.781607 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9jbk\" (UniqueName: \"kubernetes.io/projected/11604680-a734-4e85-81e3-8dc215a6fd4a-kube-api-access-k9jbk\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.782266 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.782246 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95c23d32-5f8b-4fbd-8e68-dd65447d578d-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-cx7cp\" (UID: \"95c23d32-5f8b-4fbd-8e68-dd65447d578d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp" Sep 11 14:16:57.783676 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.783658 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95c23d32-5f8b-4fbd-8e68-dd65447d578d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-cx7cp\" (UID: \"95c23d32-5f8b-4fbd-8e68-dd65447d578d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp" Sep 11 14:16:57.826265 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.826226 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cclbb\" (UniqueName: \"kubernetes.io/projected/95c23d32-5f8b-4fbd-8e68-dd65447d578d-kube-api-access-cclbb\") pod \"openshift-state-metrics-67dcf8c979-cx7cp\" (UID: \"95c23d32-5f8b-4fbd-8e68-dd65447d578d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp" Sep 11 14:16:57.882232 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.882199 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/11604680-a734-4e85-81e3-8dc215a6fd4a-root\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.882436 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.882257 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11604680-a734-4e85-81e3-8dc215a6fd4a-sys\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.882436 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.882272 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-tls\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.882436 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.882303 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11604680-a734-4e85-81e3-8dc215a6fd4a-metrics-client-ca\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.882436 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.882306 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/11604680-a734-4e85-81e3-8dc215a6fd4a-root\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.882436 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.882320 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11604680-a734-4e85-81e3-8dc215a6fd4a-sys\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.882436 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.882328 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.882801 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.882462 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.882801 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.882503 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-wtmp\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.882801 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.882539 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-textfile\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.882801 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.882564 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9jbk\" (UniqueName: \"kubernetes.io/projected/11604680-a734-4e85-81e3-8dc215a6fd4a-kube-api-access-k9jbk\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.882801 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.882610 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-wtmp\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.882998 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.882808 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-textfile\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.882998 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.882902 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11604680-a734-4e85-81e3-8dc215a6fd4a-metrics-client-ca\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:57.926548 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:57.926481 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9jbk\" (UniqueName: \"kubernetes.io/projected/11604680-a734-4e85-81e3-8dc215a6fd4a-kube-api-access-k9jbk\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:58.036034 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.035995 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:58.230762 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.230675 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:58.285854 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.285822 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95c23d32-5f8b-4fbd-8e68-dd65447d578d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-cx7cp\" (UID: \"95c23d32-5f8b-4fbd-8e68-dd65447d578d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp" Sep 11 14:16:58.288139 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.288115 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95c23d32-5f8b-4fbd-8e68-dd65447d578d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-cx7cp\" (UID: \"95c23d32-5f8b-4fbd-8e68-dd65447d578d\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp" Sep 11 14:16:58.365446 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.365396 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:58.449697 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.449656 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp" Sep 11 14:16:58.481156 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.481084 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:58.647427 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.647387 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:58.689114 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.689085 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs\") pod \"network-metrics-daemon-s2sfs\" (UID: \"05960fb2-54e8-48b2-852b-caabc3c413ca\") " pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:16:58.691427 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.691392 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05960fb2-54e8-48b2-852b-caabc3c413ca-metrics-certs\") pod \"network-metrics-daemon-s2sfs\" (UID: \"05960fb2-54e8-48b2-852b-caabc3c413ca\") " pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:16:58.793691 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.793621 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp"] Sep 11 14:16:58.796720 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:16:58.796691 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c23d32_5f8b_4fbd_8e68_dd65447d578d.slice/crio-b91b467eed0cd284c7f6aef66bb4f600fcc3e4133600499b3f052a9a7306bcca WatchSource:0}: Error finding container b91b467eed0cd284c7f6aef66bb4f600fcc3e4133600499b3f052a9a7306bcca: Status 404 returned error can't find the container with id b91b467eed0cd284c7f6aef66bb4f600fcc3e4133600499b3f052a9a7306bcca Sep 11 14:16:58.804863 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.804832 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:58.843774 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.843747 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tsp77\"" Sep 11 14:16:58.843919 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.843894 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lrm6v\"" Sep 11 14:16:58.852102 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.852085 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2sfs" Sep 11 14:16:58.854517 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.854497 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 14:16:58.858166 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.858150 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.865250 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.865230 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 11 14:16:58.866963 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.866629 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4dwbv\"" Sep 11 14:16:58.866963 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.866732 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 11 14:16:58.866963 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.866786 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 11 14:16:58.866963 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.866793 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 11 14:16:58.866963 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.866864 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 11 14:16:58.866963 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.866948 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 11 14:16:58.871586 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.871561 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 11 14:16:58.878201 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.878181 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 11 14:16:58.881517 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.881496 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 11 14:16:58.883026 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:58.882995 2553 configmap.go:193] Couldn't get configMap openshift-monitoring/node-exporter-accelerators-collector-config: failed to sync configmap cache: timed out waiting for the condition Sep 11 14:16:58.883026 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:58.883023 2553 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: failed to sync secret cache: timed out waiting for the condition Sep 11 14:16:58.883178 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:58.883060 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-accelerators-collector-config podName:11604680-a734-4e85-81e3-8dc215a6fd4a nodeName:}" failed. No retries permitted until 2026-09-11 14:16:59.383044292 +0000 UTC m=+129.997150614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-accelerators-collector-config" (UniqueName: "kubernetes.io/configmap/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-accelerators-collector-config") pod "node-exporter-kgzrk" (UID: "11604680-a734-4e85-81e3-8dc215a6fd4a") : failed to sync configmap cache: timed out waiting for the condition Sep 11 14:16:58.883178 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:58.883003 2553 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-kube-rbac-proxy-config: failed to sync secret cache: timed out waiting for the condition Sep 11 14:16:58.883178 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:58.883091 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-tls podName:11604680-a734-4e85-81e3-8dc215a6fd4a nodeName:}" failed. No retries permitted until 2026-09-11 14:16:59.383073469 +0000 UTC m=+129.997179808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-tls") pod "node-exporter-kgzrk" (UID: "11604680-a734-4e85-81e3-8dc215a6fd4a") : failed to sync secret cache: timed out waiting for the condition Sep 11 14:16:58.883178 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:16:58.883110 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-kube-rbac-proxy-config podName:11604680-a734-4e85-81e3-8dc215a6fd4a nodeName:}" failed. No retries permitted until 2026-09-11 14:16:59.383101128 +0000 UTC m=+129.997207458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-kube-rbac-proxy-config" (UniqueName: "kubernetes.io/secret/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-kube-rbac-proxy-config") pod "node-exporter-kgzrk" (UID: "11604680-a734-4e85-81e3-8dc215a6fd4a") : failed to sync secret cache: timed out waiting for the condition Sep 11 14:16:58.890753 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.890704 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35b3de7c-77d1-4b82-9a03-c846fb066bda-web-config\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.890870 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.890760 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35b3de7c-77d1-4b82-9a03-c846fb066bda-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.890870 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.890800 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxkvb\" (UniqueName: \"kubernetes.io/projected/35b3de7c-77d1-4b82-9a03-c846fb066bda-kube-api-access-mxkvb\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.890870 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.890832 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35b3de7c-77d1-4b82-9a03-c846fb066bda-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.891025 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.890872 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35b3de7c-77d1-4b82-9a03-c846fb066bda-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.891025 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.890924 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35b3de7c-77d1-4b82-9a03-c846fb066bda-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.891025 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.890987 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35b3de7c-77d1-4b82-9a03-c846fb066bda-config-out\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.891025 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.891019 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35b3de7c-77d1-4b82-9a03-c846fb066bda-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.891227 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.891057 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35b3de7c-77d1-4b82-9a03-c846fb066bda-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.891227 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.891108 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35b3de7c-77d1-4b82-9a03-c846fb066bda-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.891227 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.891145 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35b3de7c-77d1-4b82-9a03-c846fb066bda-config-volume\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.891227 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.891166 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35b3de7c-77d1-4b82-9a03-c846fb066bda-tls-assets\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.891227 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.891190 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35b3de7c-77d1-4b82-9a03-c846fb066bda-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.932768 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.932735 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 14:16:58.991789 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.991756 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35b3de7c-77d1-4b82-9a03-c846fb066bda-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.991955 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.991795 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35b3de7c-77d1-4b82-9a03-c846fb066bda-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.991955 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.991832 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35b3de7c-77d1-4b82-9a03-c846fb066bda-config-volume\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.991955 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.991857 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35b3de7c-77d1-4b82-9a03-c846fb066bda-tls-assets\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.991955 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.991880 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35b3de7c-77d1-4b82-9a03-c846fb066bda-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.991955 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.991923 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35b3de7c-77d1-4b82-9a03-c846fb066bda-web-config\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.992209 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.991962 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35b3de7c-77d1-4b82-9a03-c846fb066bda-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.992209 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.992005 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxkvb\" (UniqueName: \"kubernetes.io/projected/35b3de7c-77d1-4b82-9a03-c846fb066bda-kube-api-access-mxkvb\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.992209 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.992042 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35b3de7c-77d1-4b82-9a03-c846fb066bda-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.992209 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.992085 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35b3de7c-77d1-4b82-9a03-c846fb066bda-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.992209 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.992112 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35b3de7c-77d1-4b82-9a03-c846fb066bda-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.992209 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.992141 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35b3de7c-77d1-4b82-9a03-c846fb066bda-config-out\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.992209 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.992167 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35b3de7c-77d1-4b82-9a03-c846fb066bda-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.992676 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.992644 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35b3de7c-77d1-4b82-9a03-c846fb066bda-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.994143 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.993034 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35b3de7c-77d1-4b82-9a03-c846fb066bda-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.994454 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.994317 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35b3de7c-77d1-4b82-9a03-c846fb066bda-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.995017 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.994975 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35b3de7c-77d1-4b82-9a03-c846fb066bda-web-config\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.995258 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.995185 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35b3de7c-77d1-4b82-9a03-c846fb066bda-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.995857 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.995507 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35b3de7c-77d1-4b82-9a03-c846fb066bda-config-out\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.995857 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.995655 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35b3de7c-77d1-4b82-9a03-c846fb066bda-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.995857 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.995743 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35b3de7c-77d1-4b82-9a03-c846fb066bda-tls-assets\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.995857 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.995815 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35b3de7c-77d1-4b82-9a03-c846fb066bda-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.996067 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.996015 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35b3de7c-77d1-4b82-9a03-c846fb066bda-config-volume\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.996355 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.996338 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35b3de7c-77d1-4b82-9a03-c846fb066bda-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:58.996481 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:58.996465 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35b3de7c-77d1-4b82-9a03-c846fb066bda-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:59.000276 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:59.000261 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 14:16:59.019540 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:59.019521 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxkvb\" (UniqueName: \"kubernetes.io/projected/35b3de7c-77d1-4b82-9a03-c846fb066bda-kube-api-access-mxkvb\") pod \"alertmanager-main-0\" (UID: \"35b3de7c-77d1-4b82-9a03-c846fb066bda\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:59.155010 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:59.154983 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s2sfs"] Sep 11 14:16:59.162097 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:16:59.162072 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05960fb2_54e8_48b2_852b_caabc3c413ca.slice/crio-182a720381ec0fb8787bb3ea1294c778330f1628b083c80fce5fd5562b8f63df WatchSource:0}: Error finding container 182a720381ec0fb8787bb3ea1294c778330f1628b083c80fce5fd5562b8f63df: Status 404 returned error can't find the container with id 182a720381ec0fb8787bb3ea1294c778330f1628b083c80fce5fd5562b8f63df Sep 11 14:16:59.192547 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:59.192523 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 14:16:59.243438 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:59.243383 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 14:16:59.243995 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:59.243971 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 14:16:59.283112 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:59.283079 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:59.396664 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:59.396625 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-tls\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:59.396852 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:59.396700 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:59.396852 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:59.396731 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:59.397267 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:59.397241 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-accelerators-collector-config\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:59.399222 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:59.399191 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:59.399352 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:59.399333 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/11604680-a734-4e85-81e3-8dc215a6fd4a-node-exporter-tls\") pod \"node-exporter-kgzrk\" (UID: \"11604680-a734-4e85-81e3-8dc215a6fd4a\") " pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:59.406701 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:59.406627 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:59.448056 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:59.448015 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kgzrk" Sep 11 14:16:59.459893 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:16:59.459861 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11604680_a734_4e85_81e3_8dc215a6fd4a.slice/crio-b290934e5a38c3fbb009b52201bb3ed23b3e901d6f8df958b68ed885a0f54a82 WatchSource:0}: Error finding container b290934e5a38c3fbb009b52201bb3ed23b3e901d6f8df958b68ed885a0f54a82: Status 404 returned error can't find the container with id b290934e5a38c3fbb009b52201bb3ed23b3e901d6f8df958b68ed885a0f54a82 Sep 11 14:16:59.503624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:59.503465 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 14:16:59.543632 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:59.543590 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kgzrk" event={"ID":"11604680-a734-4e85-81e3-8dc215a6fd4a","Type":"ContainerStarted","Data":"b290934e5a38c3fbb009b52201bb3ed23b3e901d6f8df958b68ed885a0f54a82"} Sep 11 14:16:59.546429 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:59.546384 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp" event={"ID":"95c23d32-5f8b-4fbd-8e68-dd65447d578d","Type":"ContainerStarted","Data":"d9f9f4b57a5b0b3167e0ceb8edadb85000f2112876c19e029afaa70dba1f71df"} Sep 11 14:16:59.546565 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:59.546456 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp" event={"ID":"95c23d32-5f8b-4fbd-8e68-dd65447d578d","Type":"ContainerStarted","Data":"f91164f66a8d097d01c64d44e5f0362e3f41c9b0d4e2c07af84a63389760ad14"} Sep 11 14:16:59.546565 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:59.546487 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp" event={"ID":"95c23d32-5f8b-4fbd-8e68-dd65447d578d","Type":"ContainerStarted","Data":"b91b467eed0cd284c7f6aef66bb4f600fcc3e4133600499b3f052a9a7306bcca"} Sep 11 14:16:59.547773 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:16:59.547746 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s2sfs" event={"ID":"05960fb2-54e8-48b2-852b-caabc3c413ca","Type":"ContainerStarted","Data":"182a720381ec0fb8787bb3ea1294c778330f1628b083c80fce5fd5562b8f63df"} Sep 11 14:16:59.687867 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:16:59.687782 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b3de7c_77d1_4b82_9a03_c846fb066bda.slice/crio-6aaa21a103431a559645da68c5146874781f7cd6aaae3a1206b4c4fdc4569293 WatchSource:0}: Error finding container 6aaa21a103431a559645da68c5146874781f7cd6aaae3a1206b4c4fdc4569293: Status 404 returned error can't find the container with id 6aaa21a103431a559645da68c5146874781f7cd6aaae3a1206b4c4fdc4569293 Sep 11 14:17:00.553514 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:00.553465 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35b3de7c-77d1-4b82-9a03-c846fb066bda","Type":"ContainerStarted","Data":"6aaa21a103431a559645da68c5146874781f7cd6aaae3a1206b4c4fdc4569293"} Sep 11 14:17:00.555802 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:00.555777 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp" event={"ID":"95c23d32-5f8b-4fbd-8e68-dd65447d578d","Type":"ContainerStarted","Data":"2f4bbdc87bdcb82867b0676588fc1fef77100d227cd8e4f2b467b5029d15b365"} Sep 11 14:17:00.596845 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:00.596791 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-cx7cp" podStartSLOduration=2.79245663 podStartE2EDuration="3.596770929s" totalImagesPullingTime="804.314299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:00.594844918 +0000 UTC m=+131.208951263" watchObservedRunningTime="2026-09-11 14:17:00.596770929 +0000 UTC m=+131.210877275" Sep 11 14:17:01.561057 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:01.561019 2553 generic.go:356] "Generic (PLEG): container finished" podID="11604680-a734-4e85-81e3-8dc215a6fd4a" containerID="c75a625d0369ecf1a2f7e21d66ccd09e628ba28ea07daf9d68eb233cacc34592" exitCode=0 Sep 11 14:17:01.561471 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:01.561104 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kgzrk" event={"ID":"11604680-a734-4e85-81e3-8dc215a6fd4a","Type":"ContainerDied","Data":"c75a625d0369ecf1a2f7e21d66ccd09e628ba28ea07daf9d68eb233cacc34592"} Sep 11 14:17:01.563015 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:01.562988 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s2sfs" event={"ID":"05960fb2-54e8-48b2-852b-caabc3c413ca","Type":"ContainerStarted","Data":"1a762cf9e25493d58db99de4f9dceae37e8d17bcb79f4b2d59bb20205496a132"} Sep 11 14:17:01.563117 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:01.563043 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s2sfs" event={"ID":"05960fb2-54e8-48b2-852b-caabc3c413ca","Type":"ContainerStarted","Data":"8e16e498c812b9dc342303905ceafd476257633d3c4fecb4f9f99d53632dd730"} Sep 11 14:17:01.564518 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:01.564494 2553 generic.go:356] "Generic (PLEG): container finished" podID="35b3de7c-77d1-4b82-9a03-c846fb066bda" containerID="acac8725893ddd7da9c809a5b79d701d72b32ea084d7dd376ca36cd089576639" exitCode=0 Sep 11 14:17:01.564615 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:01.564581 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35b3de7c-77d1-4b82-9a03-c846fb066bda","Type":"ContainerDied","Data":"acac8725893ddd7da9c809a5b79d701d72b32ea084d7dd376ca36cd089576639"} Sep 11 14:17:01.638914 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:01.638868 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-s2sfs" podStartSLOduration=129.667890667 podStartE2EDuration="2m11.638852609s" totalImagesPullingTime="1.970961942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:01.638231816 +0000 UTC m=+132.252338176" watchObservedRunningTime="2026-09-11 14:17:01.638852609 +0000 UTC m=+132.252958953" Sep 11 14:17:01.777871 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:01.777834 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:01.943942 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:01.943907 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:02.101399 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:02.101316 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:02.211192 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:02.211101 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:02.572176 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:02.572080 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kgzrk" event={"ID":"11604680-a734-4e85-81e3-8dc215a6fd4a","Type":"ContainerStarted","Data":"1815920f8a4c2f975f368176c52a76511a72f135557836433d2cb6ac498ab0a4"} Sep 11 14:17:02.572176 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:02.572153 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kgzrk" event={"ID":"11604680-a734-4e85-81e3-8dc215a6fd4a","Type":"ContainerStarted","Data":"ed302cc4978c145dc2f2e40af1d35abf7b5ac57c0f285e35cd00461bb7a99cdc"} Sep 11 14:17:02.606713 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:02.606673 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:02.650196 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:02.650143 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kgzrk" podStartSLOduration=4.216627539 podStartE2EDuration="5.650128183s" totalImagesPullingTime="1.365174842s" totalInitContainerRuntime="68.325802ms" isStatefulPod=true podCreationTimestamp="2026-09-11 14:16:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:02.64703582 +0000 UTC m=+133.261142191" watchObservedRunningTime="2026-09-11 14:17:02.650128183 +0000 UTC m=+133.264234524" Sep 11 14:17:02.763233 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:02.763191 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:03.579841 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:03.579804 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35b3de7c-77d1-4b82-9a03-c846fb066bda","Type":"ContainerStarted","Data":"0972b2e5cd23a854b011b5a36cfab9fe7316eaef43ebe4f3f3177e054dc89aa2"} Sep 11 14:17:03.580217 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:03.579897 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35b3de7c-77d1-4b82-9a03-c846fb066bda","Type":"ContainerStarted","Data":"1c3c792b9fdf9ca8908a55214f3ff1ff6f48b4784feba660738173fb386c1e2c"} Sep 11 14:17:03.580217 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:03.579930 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35b3de7c-77d1-4b82-9a03-c846fb066bda","Type":"ContainerStarted","Data":"6f6d10d39fb94fc579b7f4a6d7ab0b64b3ee971de1c0cb855d41de81d550ebc0"} Sep 11 14:17:03.580217 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:03.579956 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35b3de7c-77d1-4b82-9a03-c846fb066bda","Type":"ContainerStarted","Data":"2fac0a9ec81e271955b407d2659c33f828d86729728c188d72b48e0c6f5aaaa8"} Sep 11 14:17:03.580217 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:03.579979 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35b3de7c-77d1-4b82-9a03-c846fb066bda","Type":"ContainerStarted","Data":"80f11a9b248e3643426c400edf403c4869e9b65a104624fa0b9803987a9a5618"} Sep 11 14:17:04.586592 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:04.586558 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35b3de7c-77d1-4b82-9a03-c846fb066bda","Type":"ContainerStarted","Data":"4a9e725ea55a6662afd64c4b71277dc530785141df43548c1599d8675051df3d"} Sep 11 14:17:04.655292 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:04.655240 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.6050515389999997 podStartE2EDuration="6.65522356s" totalImagesPullingTime="3.026582609s" totalInitContainerRuntime="23.589412ms" isStatefulPod=false podCreationTimestamp="2026-09-11 14:16:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:04.653189762 +0000 UTC m=+135.267296100" watchObservedRunningTime="2026-09-11 14:17:04.65522356 +0000 UTC m=+135.269329943" Sep 11 14:17:05.028502 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:05.028464 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:05.149575 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:05.149534 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:09.399431 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:09.399373 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:09.567336 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:09.567303 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:09.699240 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:09.699132 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:14.019616 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:14.019582 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:14.163947 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:14.163891 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:14.265530 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:14.265497 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:15.363408 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:15.363355 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:15.473510 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:15.473475 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:16.995697 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:16.995658 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:17.230699 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:17.230669 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:22.098875 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.098813 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" podUID="f32b4da3-04be-4cf9-94b9-03e8ab111b69" containerName="registry" containerID="cri-o://76d495a01055ba3bc465eb7927128defd89d37b1641137b525dc10fb6abcae50" gracePeriod=30 Sep 11 14:17:22.334482 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.334454 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:17:22.370188 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.370123 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f32b4da3-04be-4cf9-94b9-03e8ab111b69-installation-pull-secrets\") pod \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " Sep 11 14:17:22.370188 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.370164 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-certificates\") pod \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " Sep 11 14:17:22.370372 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.370201 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f32b4da3-04be-4cf9-94b9-03e8ab111b69-trusted-ca\") pod \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " Sep 11 14:17:22.370372 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.370230 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f32b4da3-04be-4cf9-94b9-03e8ab111b69-image-registry-private-configuration\") pod \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " Sep 11 14:17:22.370372 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.370257 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f32b4da3-04be-4cf9-94b9-03e8ab111b69-ca-trust-extracted\") pod \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " Sep 11 14:17:22.370372 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.370285 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls\") pod \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " Sep 11 14:17:22.370372 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.370320 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ltmz\" (UniqueName: \"kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-kube-api-access-5ltmz\") pod \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " Sep 11 14:17:22.370646 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.370373 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-bound-sa-token\") pod \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\" (UID: \"f32b4da3-04be-4cf9-94b9-03e8ab111b69\") " Sep 11 14:17:22.370709 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.370642 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f32b4da3-04be-4cf9-94b9-03e8ab111b69" (UID: "f32b4da3-04be-4cf9-94b9-03e8ab111b69"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:17:22.370709 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.370646 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f32b4da3-04be-4cf9-94b9-03e8ab111b69-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f32b4da3-04be-4cf9-94b9-03e8ab111b69" (UID: "f32b4da3-04be-4cf9-94b9-03e8ab111b69"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:17:22.372561 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.372535 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f32b4da3-04be-4cf9-94b9-03e8ab111b69-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f32b4da3-04be-4cf9-94b9-03e8ab111b69" (UID: "f32b4da3-04be-4cf9-94b9-03e8ab111b69"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:17:22.372867 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.372846 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f32b4da3-04be-4cf9-94b9-03e8ab111b69-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f32b4da3-04be-4cf9-94b9-03e8ab111b69" (UID: "f32b4da3-04be-4cf9-94b9-03e8ab111b69"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:17:22.373063 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.373036 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-kube-api-access-5ltmz" (OuterVolumeSpecName: "kube-api-access-5ltmz") pod "f32b4da3-04be-4cf9-94b9-03e8ab111b69" (UID: "f32b4da3-04be-4cf9-94b9-03e8ab111b69"). InnerVolumeSpecName "kube-api-access-5ltmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:17:22.373134 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.373081 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f32b4da3-04be-4cf9-94b9-03e8ab111b69" (UID: "f32b4da3-04be-4cf9-94b9-03e8ab111b69"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:17:22.373224 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.373205 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f32b4da3-04be-4cf9-94b9-03e8ab111b69" (UID: "f32b4da3-04be-4cf9-94b9-03e8ab111b69"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:17:22.379544 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.379518 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32b4da3-04be-4cf9-94b9-03e8ab111b69-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f32b4da3-04be-4cf9-94b9-03e8ab111b69" (UID: "f32b4da3-04be-4cf9-94b9-03e8ab111b69"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:17:22.471794 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.471753 2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f32b4da3-04be-4cf9-94b9-03e8ab111b69-trusted-ca\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:17:22.471794 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.471788 2553 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f32b4da3-04be-4cf9-94b9-03e8ab111b69-image-registry-private-configuration\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:17:22.471794 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.471800 2553 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f32b4da3-04be-4cf9-94b9-03e8ab111b69-ca-trust-extracted\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:17:22.472043 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.471810 2553 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-tls\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:17:22.472043 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.471819 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ltmz\" (UniqueName: \"kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-kube-api-access-5ltmz\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:17:22.472043 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.471827 2553 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f32b4da3-04be-4cf9-94b9-03e8ab111b69-bound-sa-token\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:17:22.472043 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.471837 2553 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f32b4da3-04be-4cf9-94b9-03e8ab111b69-installation-pull-secrets\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:17:22.472043 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.471845 2553 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f32b4da3-04be-4cf9-94b9-03e8ab111b69-registry-certificates\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:17:22.652862 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.652826 2553 generic.go:356] "Generic (PLEG): container finished" podID="f32b4da3-04be-4cf9-94b9-03e8ab111b69" containerID="76d495a01055ba3bc465eb7927128defd89d37b1641137b525dc10fb6abcae50" exitCode=0 Sep 11 14:17:22.653045 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.652881 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" Sep 11 14:17:22.653045 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.652910 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" event={"ID":"f32b4da3-04be-4cf9-94b9-03e8ab111b69","Type":"ContainerDied","Data":"76d495a01055ba3bc465eb7927128defd89d37b1641137b525dc10fb6abcae50"} Sep 11 14:17:22.653045 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.652962 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74c56b68bf-b92lx" event={"ID":"f32b4da3-04be-4cf9-94b9-03e8ab111b69","Type":"ContainerDied","Data":"9f31de9fe2e6a9b521228fb5f9b22e3a09631ab3a7b3e58b2058c26076dd0a08"} Sep 11 14:17:22.653045 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.652982 2553 scope.go:117] "RemoveContainer" containerID="76d495a01055ba3bc465eb7927128defd89d37b1641137b525dc10fb6abcae50" Sep 11 14:17:22.662126 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.662108 2553 scope.go:117] "RemoveContainer" containerID="76d495a01055ba3bc465eb7927128defd89d37b1641137b525dc10fb6abcae50" Sep 11 14:17:22.662388 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:17:22.662366 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76d495a01055ba3bc465eb7927128defd89d37b1641137b525dc10fb6abcae50\": container with ID starting with 76d495a01055ba3bc465eb7927128defd89d37b1641137b525dc10fb6abcae50 not found: ID does not exist" containerID="76d495a01055ba3bc465eb7927128defd89d37b1641137b525dc10fb6abcae50" Sep 11 14:17:22.662485 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.662400 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76d495a01055ba3bc465eb7927128defd89d37b1641137b525dc10fb6abcae50"} err="failed to get container status \"76d495a01055ba3bc465eb7927128defd89d37b1641137b525dc10fb6abcae50\": rpc error: code = NotFound desc = could not find container \"76d495a01055ba3bc465eb7927128defd89d37b1641137b525dc10fb6abcae50\": container with ID starting with 76d495a01055ba3bc465eb7927128defd89d37b1641137b525dc10fb6abcae50 not found: ID does not exist" Sep 11 14:17:22.682517 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.682496 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74c56b68bf-b92lx"] Sep 11 14:17:22.690953 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:22.690930 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-74c56b68bf-b92lx"] Sep 11 14:17:23.910860 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:23.910828 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f32b4da3-04be-4cf9-94b9-03e8ab111b69" path="/var/lib/kubelet/pods/f32b4da3-04be-4cf9-94b9-03e8ab111b69/volumes" Sep 11 14:17:44.122471 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:44.122387 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:44.221383 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:44.221346 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:49.209441 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:17:49.209392 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:57.347188 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:18:57.347152 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:57.464779 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:18:57.464747 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:49.815711 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:19:49.815676 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmqjh_6b2452d9-1730-458c-b380-dbcfd9e5be19/ovn-acl-logging/0.log" Sep 11 14:19:49.820242 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:19:49.820212 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmqjh_6b2452d9-1730-458c-b380-dbcfd9e5be19/ovn-acl-logging/0.log" Sep 11 14:19:49.829106 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:19:49.829068 2553 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 14:19:58.204866 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:19:58.204806 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:30.822697 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:20:30.822656 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:48.588823 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:21:48.588776 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:02.735468 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:02.735409 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:02.831013 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:02.830963 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:05.834300 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:05.834260 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:05.933265 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:05.933222 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:06.092278 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:06.092181 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:06.193458 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:06.193403 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:07.996393 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:07.996359 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6486f4694f-x8bc4"] Sep 11 14:22:07.996783 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:07.996678 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f32b4da3-04be-4cf9-94b9-03e8ab111b69" containerName="registry" Sep 11 14:22:07.996783 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:07.996690 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f32b4da3-04be-4cf9-94b9-03e8ab111b69" containerName="registry" Sep 11 14:22:07.996783 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:07.996749 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f32b4da3-04be-4cf9-94b9-03e8ab111b69" containerName="registry" Sep 11 14:22:07.999626 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:07.999611 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.010938 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.010915 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 11 14:22:08.011040 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.010923 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 11 14:22:08.011170 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.011156 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 11 14:22:08.011214 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.011169 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-hdk8j\"" Sep 11 14:22:08.011473 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.011458 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 11 14:22:08.011514 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.011506 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 11 14:22:08.026666 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.026647 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 11 14:22:08.068128 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.068098 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6486f4694f-x8bc4"] Sep 11 14:22:08.078760 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.078736 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16b90586-3aae-43c1-8b1b-40a81db48e7e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.078886 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.078773 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16b90586-3aae-43c1-8b1b-40a81db48e7e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.078886 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.078795 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/16b90586-3aae-43c1-8b1b-40a81db48e7e-telemeter-client-tls\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.078971 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.078881 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16b90586-3aae-43c1-8b1b-40a81db48e7e-serving-certs-ca-bundle\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.078971 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.078917 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/16b90586-3aae-43c1-8b1b-40a81db48e7e-federate-client-tls\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.078971 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.078940 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16b90586-3aae-43c1-8b1b-40a81db48e7e-metrics-client-ca\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.078971 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.078955 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/16b90586-3aae-43c1-8b1b-40a81db48e7e-secret-telemeter-client\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.079109 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.078993 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcp9x\" (UniqueName: \"kubernetes.io/projected/16b90586-3aae-43c1-8b1b-40a81db48e7e-kube-api-access-lcp9x\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.179604 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.179570 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16b90586-3aae-43c1-8b1b-40a81db48e7e-metrics-client-ca\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.179604 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.179605 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/16b90586-3aae-43c1-8b1b-40a81db48e7e-secret-telemeter-client\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.179803 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.179630 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcp9x\" (UniqueName: \"kubernetes.io/projected/16b90586-3aae-43c1-8b1b-40a81db48e7e-kube-api-access-lcp9x\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.179803 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.179668 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16b90586-3aae-43c1-8b1b-40a81db48e7e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.179803 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.179706 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16b90586-3aae-43c1-8b1b-40a81db48e7e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.179803 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.179737 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/16b90586-3aae-43c1-8b1b-40a81db48e7e-telemeter-client-tls\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.179803 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.179776 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16b90586-3aae-43c1-8b1b-40a81db48e7e-serving-certs-ca-bundle\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.180049 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.179814 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/16b90586-3aae-43c1-8b1b-40a81db48e7e-federate-client-tls\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.180981 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.180865 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16b90586-3aae-43c1-8b1b-40a81db48e7e-metrics-client-ca\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.181124 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.181004 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16b90586-3aae-43c1-8b1b-40a81db48e7e-serving-certs-ca-bundle\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.181621 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.181594 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16b90586-3aae-43c1-8b1b-40a81db48e7e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.186635 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.182748 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/16b90586-3aae-43c1-8b1b-40a81db48e7e-federate-client-tls\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.186635 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.182815 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/16b90586-3aae-43c1-8b1b-40a81db48e7e-secret-telemeter-client\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.186635 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.182868 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16b90586-3aae-43c1-8b1b-40a81db48e7e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.186635 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.182879 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/16b90586-3aae-43c1-8b1b-40a81db48e7e-telemeter-client-tls\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.189920 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.189899 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcp9x\" (UniqueName: \"kubernetes.io/projected/16b90586-3aae-43c1-8b1b-40a81db48e7e-kube-api-access-lcp9x\") pod \"telemeter-client-6486f4694f-x8bc4\" (UID: \"16b90586-3aae-43c1-8b1b-40a81db48e7e\") " pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.307968 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.307885 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" Sep 11 14:22:08.602714 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.602635 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6486f4694f-x8bc4"] Sep 11 14:22:08.605828 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:22:08.605799 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b90586_3aae_43c1_8b1b_40a81db48e7e.slice/crio-b430aad85986a5fa95abeb493a28ef4d36e42223dbddca72951633d83e177221 WatchSource:0}: Error finding container b430aad85986a5fa95abeb493a28ef4d36e42223dbddca72951633d83e177221: Status 404 returned error can't find the container with id b430aad85986a5fa95abeb493a28ef4d36e42223dbddca72951633d83e177221 Sep 11 14:22:08.608032 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.608014 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:22:08.646280 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:08.646248 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" event={"ID":"16b90586-3aae-43c1-8b1b-40a81db48e7e","Type":"ContainerStarted","Data":"b430aad85986a5fa95abeb493a28ef4d36e42223dbddca72951633d83e177221"} Sep 11 14:22:09.172787 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.172757 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 14:22:09.176529 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.176505 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.183490 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.183468 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 11 14:22:09.184954 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.184780 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 11 14:22:09.184954 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.184833 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 11 14:22:09.184954 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.184942 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5bkgz\"" Sep 11 14:22:09.185558 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.185527 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 11 14:22:09.185742 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.185722 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 11 14:22:09.185834 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.185758 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 11 14:22:09.185834 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.185775 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 11 14:22:09.185834 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.185721 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 11 14:22:09.186000 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.185921 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d1d1gjs5tuvpg\"" Sep 11 14:22:09.186116 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.186093 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 11 14:22:09.186221 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.186123 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 11 14:22:09.186567 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.186548 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 11 14:22:09.186960 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.186942 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 11 14:22:09.187062 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.186966 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 11 14:22:09.191704 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.191684 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 11 14:22:09.197717 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.197696 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 11 14:22:09.205290 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.205260 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:09.212860 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.212835 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 14:22:09.288124 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.288090 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7637927e-8707-48f3-afaf-d64d2c4c4b38-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.288124 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.288119 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7637927e-8707-48f3-afaf-d64d2c4c4b38-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.288332 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.288143 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-config\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.288332 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.288186 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.288332 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.288255 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-web-config\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.288332 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.288285 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.288332 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.288329 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.288543 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.288349 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7637927e-8707-48f3-afaf-d64d2c4c4b38-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.288543 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.288367 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7637927e-8707-48f3-afaf-d64d2c4c4b38-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.288543 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.288383 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7637927e-8707-48f3-afaf-d64d2c4c4b38-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.288543 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.288433 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.288543 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.288461 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsjqb\" (UniqueName: \"kubernetes.io/projected/7637927e-8707-48f3-afaf-d64d2c4c4b38-kube-api-access-bsjqb\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.288543 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.288484 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7637927e-8707-48f3-afaf-d64d2c4c4b38-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.288543 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.288500 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7637927e-8707-48f3-afaf-d64d2c4c4b38-config-out\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.288543 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.288523 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.288809 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.288547 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7637927e-8707-48f3-afaf-d64d2c4c4b38-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.288809 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.288563 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.288809 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.288577 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7637927e-8707-48f3-afaf-d64d2c4c4b38-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.288809 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.288641 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.288809 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.288672 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7637927e-8707-48f3-afaf-d64d2c4c4b38-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.389051 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.389011 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7637927e-8707-48f3-afaf-d64d2c4c4b38-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.389238 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.389062 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7637927e-8707-48f3-afaf-d64d2c4c4b38-config-out\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.389238 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.389085 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.389238 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.389110 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7637927e-8707-48f3-afaf-d64d2c4c4b38-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.389408 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.389272 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.389408 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.389308 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7637927e-8707-48f3-afaf-d64d2c4c4b38-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.389408 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.389330 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.389408 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.389362 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7637927e-8707-48f3-afaf-d64d2c4c4b38-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.389408 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.389391 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7637927e-8707-48f3-afaf-d64d2c4c4b38-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.389408 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.389429 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7637927e-8707-48f3-afaf-d64d2c4c4b38-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.389722 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.389460 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-config\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.389722 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.389489 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.389722 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.389529 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-web-config\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.389722 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.389568 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.389722 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.389600 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.389722 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.389624 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7637927e-8707-48f3-afaf-d64d2c4c4b38-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.389722 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.389654 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7637927e-8707-48f3-afaf-d64d2c4c4b38-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.389722 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.389685 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7637927e-8707-48f3-afaf-d64d2c4c4b38-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.389722 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.389720 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.390130 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.389748 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsjqb\" (UniqueName: \"kubernetes.io/projected/7637927e-8707-48f3-afaf-d64d2c4c4b38-kube-api-access-bsjqb\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.392010 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.391155 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7637927e-8707-48f3-afaf-d64d2c4c4b38-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.392010 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.391388 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7637927e-8707-48f3-afaf-d64d2c4c4b38-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.392010 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.391447 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7637927e-8707-48f3-afaf-d64d2c4c4b38-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.392010 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.391721 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7637927e-8707-48f3-afaf-d64d2c4c4b38-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.392320 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.392019 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7637927e-8707-48f3-afaf-d64d2c4c4b38-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.392320 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.392122 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7637927e-8707-48f3-afaf-d64d2c4c4b38-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.392670 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.392552 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.392791 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.392772 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7637927e-8707-48f3-afaf-d64d2c4c4b38-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.393750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.393137 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7637927e-8707-48f3-afaf-d64d2c4c4b38-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.394275 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.394254 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7637927e-8707-48f3-afaf-d64d2c4c4b38-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.394402 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.394309 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7637927e-8707-48f3-afaf-d64d2c4c4b38-config-out\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.394764 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.394516 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.394994 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.394973 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.395519 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.395492 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.395702 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.395678 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.396306 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.396279 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-web-config\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.396489 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.396471 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.396853 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.396838 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-config\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.397139 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.397124 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7637927e-8707-48f3-afaf-d64d2c4c4b38-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.405505 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.405484 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsjqb\" (UniqueName: \"kubernetes.io/projected/7637927e-8707-48f3-afaf-d64d2c4c4b38-kube-api-access-bsjqb\") pod \"prometheus-k8s-0\" (UID: \"7637927e-8707-48f3-afaf-d64d2c4c4b38\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.488711 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.488635 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:09.790759 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:09.790733 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 11 14:22:09.792571 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:22:09.792542 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7637927e_8707_48f3_afaf_d64d2c4c4b38.slice/crio-2f09d74d9b2944df5de748fe010a03bbc9980490258c78ca923d298f89d84228 WatchSource:0}: Error finding container 2f09d74d9b2944df5de748fe010a03bbc9980490258c78ca923d298f89d84228: Status 404 returned error can't find the container with id 2f09d74d9b2944df5de748fe010a03bbc9980490258c78ca923d298f89d84228 Sep 11 14:22:10.655599 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:10.655562 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" event={"ID":"16b90586-3aae-43c1-8b1b-40a81db48e7e","Type":"ContainerStarted","Data":"1ecab72cd0aea46dbad9cb747f204a3961d1c2ce50ebe0850c42243c51efcc83"} Sep 11 14:22:10.656154 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:10.655659 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" event={"ID":"16b90586-3aae-43c1-8b1b-40a81db48e7e","Type":"ContainerStarted","Data":"85b0e6a24e8ae4bb75c21c251e5476be87567006b2ef4365ece2a218aac82439"} Sep 11 14:22:10.656154 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:10.655683 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" event={"ID":"16b90586-3aae-43c1-8b1b-40a81db48e7e","Type":"ContainerStarted","Data":"41a1148a1ca0d9ec2f5f1434a4d47e51975e64a5895d2f76bd00f831ecfb6c11"} Sep 11 14:22:10.657248 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:10.657222 2553 generic.go:356] "Generic (PLEG): container finished" podID="7637927e-8707-48f3-afaf-d64d2c4c4b38" containerID="bd0e553d0774e9fba9a60154476baf7e5e4ecdebcb37c835b50783ff976a008d" exitCode=0 Sep 11 14:22:10.657368 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:10.657260 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7637927e-8707-48f3-afaf-d64d2c4c4b38","Type":"ContainerDied","Data":"bd0e553d0774e9fba9a60154476baf7e5e4ecdebcb37c835b50783ff976a008d"} Sep 11 14:22:10.657368 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:10.657319 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7637927e-8707-48f3-afaf-d64d2c4c4b38","Type":"ContainerStarted","Data":"2f09d74d9b2944df5de748fe010a03bbc9980490258c78ca923d298f89d84228"} Sep 11 14:22:10.690357 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:10.690302 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6486f4694f-x8bc4" podStartSLOduration=1.8489038020000002 podStartE2EDuration="3.690284955s" totalImagesPullingTime="1.841381153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:22:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:22:10.689008909 +0000 UTC m=+441.303115254" watchObservedRunningTime="2026-09-11 14:22:10.690284955 +0000 UTC m=+441.304391298" Sep 11 14:22:10.913566 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:10.913535 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:11.056667 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:11.056632 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:14.676870 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:14.676834 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7637927e-8707-48f3-afaf-d64d2c4c4b38","Type":"ContainerStarted","Data":"f9e7ae4ccc22aefa0a1edf921b993d211a868779d9090983b935877ef690ee94"} Sep 11 14:22:14.677320 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:14.676914 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7637927e-8707-48f3-afaf-d64d2c4c4b38","Type":"ContainerStarted","Data":"6c96e6ec9340c10ce97314e045ba6fe27de5d5e03f964293e8ff634607ed2573"} Sep 11 14:22:16.689230 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:16.689192 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7637927e-8707-48f3-afaf-d64d2c4c4b38","Type":"ContainerStarted","Data":"f9fc473f25416a86473488eb7b1b793e320c5c6328236ea00bd4fd3136c4b870"} Sep 11 14:22:16.689602 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:16.689258 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7637927e-8707-48f3-afaf-d64d2c4c4b38","Type":"ContainerStarted","Data":"86480531d3730cb06e0d0175456fa002111d7761523fdd1452bd545c0d6dfc30"} Sep 11 14:22:16.689602 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:16.689292 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7637927e-8707-48f3-afaf-d64d2c4c4b38","Type":"ContainerStarted","Data":"ca7b37c6133555aa59b771a37cf3a9b795c545ae5f081f7122071d8893d54fe9"} Sep 11 14:22:16.689602 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:16.689320 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7637927e-8707-48f3-afaf-d64d2c4c4b38","Type":"ContainerStarted","Data":"f7950d3372bc9d439924501d0bfc194cfa5a3dfbcc8e3b5167ddf24f17118e27"} Sep 11 14:22:16.734977 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:16.734911 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.573532572 podStartE2EDuration="7.734890174s" totalImagesPullingTime="5.139984783s" totalInitContainerRuntime="21.372819ms" isStatefulPod=false podCreationTimestamp="2026-09-11 14:22:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:22:16.732876648 +0000 UTC m=+447.346983031" watchObservedRunningTime="2026-09-11 14:22:16.734890174 +0000 UTC m=+447.348996518" Sep 11 14:22:19.489509 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:19.489467 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:22:21.621817 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:21.621785 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:21.812566 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:21.812529 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:38.626717 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:38.626684 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:38.719363 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:22:38.719328 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:09.489493 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:23:09.489446 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:23:09.508870 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:23:09.508843 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:23:09.912063 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:23:09.912022 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 11 14:23:19.144503 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:23:19.144467 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:19.357450 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:23:19.357396 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:31.669728 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:23:31.669689 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:31.812529 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:23:31.812480 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:36.834253 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:23:36.834201 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:38.350390 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:23:38.350355 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:42.956668 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:23:42.956582 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:43.096561 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:23:43.096529 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:44.123179 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:23:44.123136 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:44.211484 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:23:44.211450 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:48.023141 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:23:48.023099 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:48.278668 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:23:48.278568 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:38.766585 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:24:38.766546 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:38.896462 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:24:38.896402 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:49.850638 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:24:49.850608 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmqjh_6b2452d9-1730-458c-b380-dbcfd9e5be19/ovn-acl-logging/0.log" Sep 11 14:24:49.851534 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:24:49.851510 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmqjh_6b2452d9-1730-458c-b380-dbcfd9e5be19/ovn-acl-logging/0.log" Sep 11 14:25:08.746977 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:08.746935 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:08.929561 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:08.929528 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:13.924400 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:13.924312 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:14.058840 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:14.058805 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:16.699532 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:16.699497 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:17.166977 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:17.166941 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:17.595806 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:17.595720 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:19.010516 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:19.010473 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:19.470224 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:19.470195 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:21.107523 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:21.107487 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:21.224495 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:21.224463 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:25.913245 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:25.913208 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:26.168600 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:26.168506 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:27.932585 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:27.932545 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:33.893483 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:33.893399 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:35.521590 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:35.521560 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:50.283046 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:50.283008 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:50.445795 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:50.445745 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:54.389363 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:54.389304 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:54.540975 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:54.540939 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:56.119948 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:56.119912 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:58.156549 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:58.156504 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:58.243018 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:25:58.242984 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:00.384720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.384685 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:00.447545 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.447511 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-55cb89464b-tncr8"] Sep 11 14:26:00.450849 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.450833 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-55cb89464b-tncr8" Sep 11 14:26:00.461977 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.461954 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 11 14:26:00.462180 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.462167 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 11 14:26:00.463399 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.463384 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 11 14:26:00.463611 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.463595 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 11 14:26:00.464889 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.464876 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mf6fd\"" Sep 11 14:26:00.466854 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.466840 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 11 14:26:00.471976 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.471955 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-55cb89464b-tncr8"] Sep 11 14:26:00.521838 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.521804 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:00.563525 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.563495 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/69decb7e-f8a7-4e03-9add-0f25b3dbb59b-manager-config\") pod \"lws-controller-manager-55cb89464b-tncr8\" (UID: \"69decb7e-f8a7-4e03-9add-0f25b3dbb59b\") " pod="openshift-lws-operator/lws-controller-manager-55cb89464b-tncr8" Sep 11 14:26:00.563685 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.563536 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69decb7e-f8a7-4e03-9add-0f25b3dbb59b-cert\") pod \"lws-controller-manager-55cb89464b-tncr8\" (UID: \"69decb7e-f8a7-4e03-9add-0f25b3dbb59b\") " pod="openshift-lws-operator/lws-controller-manager-55cb89464b-tncr8" Sep 11 14:26:00.563685 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.563569 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/69decb7e-f8a7-4e03-9add-0f25b3dbb59b-metrics-cert\") pod \"lws-controller-manager-55cb89464b-tncr8\" (UID: \"69decb7e-f8a7-4e03-9add-0f25b3dbb59b\") " pod="openshift-lws-operator/lws-controller-manager-55cb89464b-tncr8" Sep 11 14:26:00.563685 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.563665 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dckqj\" (UniqueName: \"kubernetes.io/projected/69decb7e-f8a7-4e03-9add-0f25b3dbb59b-kube-api-access-dckqj\") pod \"lws-controller-manager-55cb89464b-tncr8\" (UID: \"69decb7e-f8a7-4e03-9add-0f25b3dbb59b\") " pod="openshift-lws-operator/lws-controller-manager-55cb89464b-tncr8" Sep 11 14:26:00.664596 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.664494 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69decb7e-f8a7-4e03-9add-0f25b3dbb59b-cert\") pod \"lws-controller-manager-55cb89464b-tncr8\" (UID: \"69decb7e-f8a7-4e03-9add-0f25b3dbb59b\") " pod="openshift-lws-operator/lws-controller-manager-55cb89464b-tncr8" Sep 11 14:26:00.664762 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.664631 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/69decb7e-f8a7-4e03-9add-0f25b3dbb59b-metrics-cert\") pod \"lws-controller-manager-55cb89464b-tncr8\" (UID: \"69decb7e-f8a7-4e03-9add-0f25b3dbb59b\") " pod="openshift-lws-operator/lws-controller-manager-55cb89464b-tncr8" Sep 11 14:26:00.664762 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.664699 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dckqj\" (UniqueName: \"kubernetes.io/projected/69decb7e-f8a7-4e03-9add-0f25b3dbb59b-kube-api-access-dckqj\") pod \"lws-controller-manager-55cb89464b-tncr8\" (UID: \"69decb7e-f8a7-4e03-9add-0f25b3dbb59b\") " pod="openshift-lws-operator/lws-controller-manager-55cb89464b-tncr8" Sep 11 14:26:00.664762 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.664737 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/69decb7e-f8a7-4e03-9add-0f25b3dbb59b-manager-config\") pod \"lws-controller-manager-55cb89464b-tncr8\" (UID: \"69decb7e-f8a7-4e03-9add-0f25b3dbb59b\") " pod="openshift-lws-operator/lws-controller-manager-55cb89464b-tncr8" Sep 11 14:26:00.665344 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.665322 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/69decb7e-f8a7-4e03-9add-0f25b3dbb59b-manager-config\") pod \"lws-controller-manager-55cb89464b-tncr8\" (UID: \"69decb7e-f8a7-4e03-9add-0f25b3dbb59b\") " pod="openshift-lws-operator/lws-controller-manager-55cb89464b-tncr8" Sep 11 14:26:00.666865 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.666844 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69decb7e-f8a7-4e03-9add-0f25b3dbb59b-cert\") pod \"lws-controller-manager-55cb89464b-tncr8\" (UID: \"69decb7e-f8a7-4e03-9add-0f25b3dbb59b\") " pod="openshift-lws-operator/lws-controller-manager-55cb89464b-tncr8" Sep 11 14:26:00.666966 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.666948 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/69decb7e-f8a7-4e03-9add-0f25b3dbb59b-metrics-cert\") pod \"lws-controller-manager-55cb89464b-tncr8\" (UID: \"69decb7e-f8a7-4e03-9add-0f25b3dbb59b\") " pod="openshift-lws-operator/lws-controller-manager-55cb89464b-tncr8" Sep 11 14:26:00.681707 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.681688 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dckqj\" (UniqueName: \"kubernetes.io/projected/69decb7e-f8a7-4e03-9add-0f25b3dbb59b-kube-api-access-dckqj\") pod \"lws-controller-manager-55cb89464b-tncr8\" (UID: \"69decb7e-f8a7-4e03-9add-0f25b3dbb59b\") " pod="openshift-lws-operator/lws-controller-manager-55cb89464b-tncr8" Sep 11 14:26:00.759559 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:00.759523 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-55cb89464b-tncr8" Sep 11 14:26:01.042033 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:01.041952 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-55cb89464b-tncr8"] Sep 11 14:26:01.044919 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:26:01.044898 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69decb7e_f8a7_4e03_9add_0f25b3dbb59b.slice/crio-e780b3fbcd73b2f07bfe910c3a527c66c4a84dd5155c00ef871528069e096749 WatchSource:0}: Error finding container e780b3fbcd73b2f07bfe910c3a527c66c4a84dd5155c00ef871528069e096749: Status 404 returned error can't find the container with id e780b3fbcd73b2f07bfe910c3a527c66c4a84dd5155c00ef871528069e096749 Sep 11 14:26:01.554494 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:01.554456 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-55cb89464b-tncr8" event={"ID":"69decb7e-f8a7-4e03-9add-0f25b3dbb59b","Type":"ContainerStarted","Data":"e780b3fbcd73b2f07bfe910c3a527c66c4a84dd5155c00ef871528069e096749"} Sep 11 14:26:03.565848 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:03.565811 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-55cb89464b-tncr8" event={"ID":"69decb7e-f8a7-4e03-9add-0f25b3dbb59b","Type":"ContainerStarted","Data":"b97fef1b020fae0d412838160907f467290ca89d4b456e3f322c4f82fea56338"} Sep 11 14:26:03.566205 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:03.565932 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-55cb89464b-tncr8" Sep 11 14:26:03.588048 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:03.587985 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-55cb89464b-tncr8" podStartSLOduration=1.18482788 podStartE2EDuration="3.587968604s" totalImagesPullingTime="2.403140724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:26:03.587924701 +0000 UTC m=+674.202031046" watchObservedRunningTime="2026-09-11 14:26:03.587968604 +0000 UTC m=+674.202074950" Sep 11 14:26:11.719730 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:11.719696 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:11.828220 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:11.828168 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:12.271192 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:12.271153 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:12.740567 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:12.740501 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:12.849218 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:12.849188 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:12.930166 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:12.930136 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr"] Sep 11 14:26:12.936612 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:12.936593 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" Sep 11 14:26:12.987435 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:26:12.987385 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-dockercfg-vsjmn\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vsjmn\"" type="*v1.Secret" Sep 11 14:26:12.988180 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:26:12.988147 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"opendatahub-operator-controller-manager-86d7f488b8-8xdsr\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="e018c54d-e91d-4b5d-997f-cf8884c3cac9" pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" Sep 11 14:26:12.988180 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:26:12.988162 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-service-cert\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" Sep 11 14:26:12.988281 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:26:12.988170 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" type="*v1.Secret" Sep 11 14:26:12.988281 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:26:12.988217 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Sep 11 14:26:12.988281 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:26:12.988244 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Sep 11 14:26:13.013486 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:13.011766 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr"] Sep 11 14:26:13.075604 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:13.075564 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e018c54d-e91d-4b5d-997f-cf8884c3cac9-webhook-cert\") pod \"opendatahub-operator-controller-manager-86d7f488b8-8xdsr\" (UID: \"e018c54d-e91d-4b5d-997f-cf8884c3cac9\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" Sep 11 14:26:13.075604 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:13.075608 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e018c54d-e91d-4b5d-997f-cf8884c3cac9-manifests\") pod \"opendatahub-operator-controller-manager-86d7f488b8-8xdsr\" (UID: \"e018c54d-e91d-4b5d-997f-cf8884c3cac9\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" Sep 11 14:26:13.075820 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:13.075633 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e018c54d-e91d-4b5d-997f-cf8884c3cac9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86d7f488b8-8xdsr\" (UID: \"e018c54d-e91d-4b5d-997f-cf8884c3cac9\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" Sep 11 14:26:13.075820 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:13.075722 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqr2g\" (UniqueName: \"kubernetes.io/projected/e018c54d-e91d-4b5d-997f-cf8884c3cac9-kube-api-access-rqr2g\") pod \"opendatahub-operator-controller-manager-86d7f488b8-8xdsr\" (UID: \"e018c54d-e91d-4b5d-997f-cf8884c3cac9\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" Sep 11 14:26:13.075820 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:13.075776 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e018c54d-e91d-4b5d-997f-cf8884c3cac9-tmp\") pod \"opendatahub-operator-controller-manager-86d7f488b8-8xdsr\" (UID: \"e018c54d-e91d-4b5d-997f-cf8884c3cac9\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" Sep 11 14:26:13.177046 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:13.177004 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e018c54d-e91d-4b5d-997f-cf8884c3cac9-webhook-cert\") pod \"opendatahub-operator-controller-manager-86d7f488b8-8xdsr\" (UID: \"e018c54d-e91d-4b5d-997f-cf8884c3cac9\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" Sep 11 14:26:13.177046 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:13.177049 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e018c54d-e91d-4b5d-997f-cf8884c3cac9-manifests\") pod \"opendatahub-operator-controller-manager-86d7f488b8-8xdsr\" (UID: \"e018c54d-e91d-4b5d-997f-cf8884c3cac9\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" Sep 11 14:26:13.177262 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:13.177076 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e018c54d-e91d-4b5d-997f-cf8884c3cac9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86d7f488b8-8xdsr\" (UID: \"e018c54d-e91d-4b5d-997f-cf8884c3cac9\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" Sep 11 14:26:13.177262 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:13.177112 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqr2g\" (UniqueName: \"kubernetes.io/projected/e018c54d-e91d-4b5d-997f-cf8884c3cac9-kube-api-access-rqr2g\") pod \"opendatahub-operator-controller-manager-86d7f488b8-8xdsr\" (UID: \"e018c54d-e91d-4b5d-997f-cf8884c3cac9\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" Sep 11 14:26:13.177262 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:13.177163 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e018c54d-e91d-4b5d-997f-cf8884c3cac9-tmp\") pod \"opendatahub-operator-controller-manager-86d7f488b8-8xdsr\" (UID: \"e018c54d-e91d-4b5d-997f-cf8884c3cac9\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" Sep 11 14:26:13.177563 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:13.177541 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e018c54d-e91d-4b5d-997f-cf8884c3cac9-manifests\") pod \"opendatahub-operator-controller-manager-86d7f488b8-8xdsr\" (UID: \"e018c54d-e91d-4b5d-997f-cf8884c3cac9\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" Sep 11 14:26:13.179315 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:13.179296 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e018c54d-e91d-4b5d-997f-cf8884c3cac9-tmp\") pod \"opendatahub-operator-controller-manager-86d7f488b8-8xdsr\" (UID: \"e018c54d-e91d-4b5d-997f-cf8884c3cac9\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" Sep 11 14:26:13.247367 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:13.247324 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:13.353966 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:13.353870 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:13.482534 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:13.482504 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:13.635362 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:13.635279 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:13.739524 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:13.739485 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:13.882224 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:13.882193 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 14:26:13.890073 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:13.890047 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e018c54d-e91d-4b5d-997f-cf8884c3cac9-webhook-cert\") pod \"opendatahub-operator-controller-manager-86d7f488b8-8xdsr\" (UID: \"e018c54d-e91d-4b5d-997f-cf8884c3cac9\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" Sep 11 14:26:13.890204 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:13.890079 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e018c54d-e91d-4b5d-997f-cf8884c3cac9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86d7f488b8-8xdsr\" (UID: \"e018c54d-e91d-4b5d-997f-cf8884c3cac9\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" Sep 11 14:26:14.077700 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:14.077664 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 14:26:14.085193 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:14.085170 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 14:26:14.090007 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:14.089983 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqr2g\" (UniqueName: \"kubernetes.io/projected/e018c54d-e91d-4b5d-997f-cf8884c3cac9-kube-api-access-rqr2g\") pod \"opendatahub-operator-controller-manager-86d7f488b8-8xdsr\" (UID: \"e018c54d-e91d-4b5d-997f-cf8884c3cac9\") " pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" Sep 11 14:26:14.242901 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:14.242821 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 14:26:14.272595 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:14.272571 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vsjmn\"" Sep 11 14:26:14.277151 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:14.277134 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" Sep 11 14:26:14.561469 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:14.561440 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr"] Sep 11 14:26:14.561979 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:26:14.561958 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode018c54d_e91d_4b5d_997f_cf8884c3cac9.slice/crio-5b62527e7a332833f97ef7682ff7a83c9304d43c35f60e3aa768f7b4e0bf21bd WatchSource:0}: Error finding container 5b62527e7a332833f97ef7682ff7a83c9304d43c35f60e3aa768f7b4e0bf21bd: Status 404 returned error can't find the container with id 5b62527e7a332833f97ef7682ff7a83c9304d43c35f60e3aa768f7b4e0bf21bd Sep 11 14:26:14.572552 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:14.572528 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-55cb89464b-tncr8" Sep 11 14:26:14.608769 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:14.608736 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" event={"ID":"e018c54d-e91d-4b5d-997f-cf8884c3cac9","Type":"ContainerStarted","Data":"5b62527e7a332833f97ef7682ff7a83c9304d43c35f60e3aa768f7b4e0bf21bd"} Sep 11 14:26:17.624718 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:17.624681 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" event={"ID":"e018c54d-e91d-4b5d-997f-cf8884c3cac9","Type":"ContainerStarted","Data":"46d94ee61b8420f7ed4cf4cc119599742c64169fce4bdc056b60fe50680078c5"} Sep 11 14:26:18.631056 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:18.631005 2553 generic.go:356] "Generic (PLEG): container finished" podID="e018c54d-e91d-4b5d-997f-cf8884c3cac9" containerID="46d94ee61b8420f7ed4cf4cc119599742c64169fce4bdc056b60fe50680078c5" exitCode=0 Sep 11 14:26:18.631463 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:18.631086 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" event={"ID":"e018c54d-e91d-4b5d-997f-cf8884c3cac9","Type":"ContainerDied","Data":"46d94ee61b8420f7ed4cf4cc119599742c64169fce4bdc056b60fe50680078c5"} Sep 11 14:26:19.637514 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:19.637399 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" event={"ID":"e018c54d-e91d-4b5d-997f-cf8884c3cac9","Type":"ContainerStarted","Data":"6311b9588221f01e7c3964a2ddb59e6b57bb129a874918476027183c0ade7f29"} Sep 11 14:26:19.637514 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:19.637503 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" Sep 11 14:26:19.668488 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:19.668425 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" podStartSLOduration=3.006875305 podStartE2EDuration="7.668389242s" totalImagesPullingTime="3.563690055s" totalInitContainerRuntime="1.097823882s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:26:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:26:19.66721742 +0000 UTC m=+690.281323766" watchObservedRunningTime="2026-09-11 14:26:19.668389242 +0000 UTC m=+690.282495588" Sep 11 14:26:23.970555 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:23.970523 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:24.052754 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:24.052716 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:30.644485 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:30.644456 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-86d7f488b8-8xdsr" Sep 11 14:26:32.928339 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:32.928296 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:33.126872 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:33.126834 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:34.521185 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:34.521152 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-769cf48dc6-74wxb"] Sep 11 14:26:34.524266 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:34.524228 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:34.530295 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:34.530277 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-769cf48dc6-74wxb" Sep 11 14:26:34.534531 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:34.534512 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 11 14:26:34.534612 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:34.534552 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2nmcq\"" Sep 11 14:26:34.535085 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:34.535067 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 14:26:34.535407 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:34.535395 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 11 14:26:34.535490 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:34.535395 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 14:26:34.551943 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:34.551910 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-769cf48dc6-74wxb"] Sep 11 14:26:34.564163 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:34.564135 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgqlj\" (UniqueName: \"kubernetes.io/projected/f40b578b-ce2a-4eca-96c9-0f88c9e90261-kube-api-access-sgqlj\") pod \"kube-auth-proxy-769cf48dc6-74wxb\" (UID: \"f40b578b-ce2a-4eca-96c9-0f88c9e90261\") " pod="openshift-ingress/kube-auth-proxy-769cf48dc6-74wxb" Sep 11 14:26:34.564284 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:34.564186 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f40b578b-ce2a-4eca-96c9-0f88c9e90261-tls-certs\") pod \"kube-auth-proxy-769cf48dc6-74wxb\" (UID: \"f40b578b-ce2a-4eca-96c9-0f88c9e90261\") " pod="openshift-ingress/kube-auth-proxy-769cf48dc6-74wxb" Sep 11 14:26:34.564284 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:34.564202 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f40b578b-ce2a-4eca-96c9-0f88c9e90261-tmp\") pod \"kube-auth-proxy-769cf48dc6-74wxb\" (UID: \"f40b578b-ce2a-4eca-96c9-0f88c9e90261\") " pod="openshift-ingress/kube-auth-proxy-769cf48dc6-74wxb" Sep 11 14:26:34.664807 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:34.664779 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f40b578b-ce2a-4eca-96c9-0f88c9e90261-tls-certs\") pod \"kube-auth-proxy-769cf48dc6-74wxb\" (UID: \"f40b578b-ce2a-4eca-96c9-0f88c9e90261\") " pod="openshift-ingress/kube-auth-proxy-769cf48dc6-74wxb" Sep 11 14:26:34.664807 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:34.664808 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f40b578b-ce2a-4eca-96c9-0f88c9e90261-tmp\") pod \"kube-auth-proxy-769cf48dc6-74wxb\" (UID: \"f40b578b-ce2a-4eca-96c9-0f88c9e90261\") " pod="openshift-ingress/kube-auth-proxy-769cf48dc6-74wxb" Sep 11 14:26:34.665065 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:34.664874 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgqlj\" (UniqueName: \"kubernetes.io/projected/f40b578b-ce2a-4eca-96c9-0f88c9e90261-kube-api-access-sgqlj\") pod \"kube-auth-proxy-769cf48dc6-74wxb\" (UID: \"f40b578b-ce2a-4eca-96c9-0f88c9e90261\") " pod="openshift-ingress/kube-auth-proxy-769cf48dc6-74wxb" Sep 11 14:26:34.667177 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:34.667144 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f40b578b-ce2a-4eca-96c9-0f88c9e90261-tmp\") pod \"kube-auth-proxy-769cf48dc6-74wxb\" (UID: \"f40b578b-ce2a-4eca-96c9-0f88c9e90261\") " pod="openshift-ingress/kube-auth-proxy-769cf48dc6-74wxb" Sep 11 14:26:34.667291 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:34.667270 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f40b578b-ce2a-4eca-96c9-0f88c9e90261-tls-certs\") pod \"kube-auth-proxy-769cf48dc6-74wxb\" (UID: \"f40b578b-ce2a-4eca-96c9-0f88c9e90261\") " pod="openshift-ingress/kube-auth-proxy-769cf48dc6-74wxb" Sep 11 14:26:34.684370 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:34.684344 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgqlj\" (UniqueName: \"kubernetes.io/projected/f40b578b-ce2a-4eca-96c9-0f88c9e90261-kube-api-access-sgqlj\") pod \"kube-auth-proxy-769cf48dc6-74wxb\" (UID: \"f40b578b-ce2a-4eca-96c9-0f88c9e90261\") " pod="openshift-ingress/kube-auth-proxy-769cf48dc6-74wxb" Sep 11 14:26:34.842357 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:34.842277 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-769cf48dc6-74wxb" Sep 11 14:26:35.173528 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:35.173498 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-769cf48dc6-74wxb"] Sep 11 14:26:35.176968 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:26:35.176946 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40b578b_ce2a_4eca_96c9_0f88c9e90261.slice/crio-94fe79cf2b2481728ebb9dc509816ee24a104e88a685f50cfc2b0b6e841a3cfe WatchSource:0}: Error finding container 94fe79cf2b2481728ebb9dc509816ee24a104e88a685f50cfc2b0b6e841a3cfe: Status 404 returned error can't find the container with id 94fe79cf2b2481728ebb9dc509816ee24a104e88a685f50cfc2b0b6e841a3cfe Sep 11 14:26:35.302575 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:35.302539 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:35.705510 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:35.705474 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-769cf48dc6-74wxb" event={"ID":"f40b578b-ce2a-4eca-96c9-0f88c9e90261","Type":"ContainerStarted","Data":"94fe79cf2b2481728ebb9dc509816ee24a104e88a685f50cfc2b0b6e841a3cfe"} Sep 11 14:26:36.589297 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:36.589196 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:36.752918 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:36.752878 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:38.565150 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:38.565118 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:38.719633 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:38.719599 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-769cf48dc6-74wxb" event={"ID":"f40b578b-ce2a-4eca-96c9-0f88c9e90261","Type":"ContainerStarted","Data":"8b637f7ff12dc600690c057a8724b85de7b7b9ff493eba3da47fede64af1a6db"} Sep 11 14:26:38.757516 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:38.757458 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-769cf48dc6-74wxb" podStartSLOduration=1.869973087 podStartE2EDuration="4.757442778s" totalImagesPullingTime="2.887469691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:26:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:26:38.756762455 +0000 UTC m=+709.370868800" watchObservedRunningTime="2026-09-11 14:26:38.757442778 +0000 UTC m=+709.371549166" Sep 11 14:26:42.565880 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:42.565794 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:42.707790 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:42.707758 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:44.541901 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:44.541838 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:44.716446 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:44.716389 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:44.874720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:44.874627 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:45.014487 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:45.014455 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:45.130597 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:45.130504 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:45.429587 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:45.429544 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:45.778908 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:45.778821 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:45.910724 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:45.910687 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:58.155274 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:58.155227 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:58.277645 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:58.277607 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:58.414756 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:58.414659 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:58.569081 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:58.569047 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:59.887347 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:26:59.887312 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:27:00.004448 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:27:00.004403 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:27:00.297410 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:27:00.297362 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:27:00.579879 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:27:00.579791 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:27:11.897847 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:27:11.897786 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:27:11.994071 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:27:11.994040 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:27:12.085897 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:27:12.085864 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:27:35.668560 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:27:35.668520 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:27:45.166344 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:27:45.166302 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:27:45.259717 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:27:45.259683 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:00.882266 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:00.882234 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:03.148628 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:03.148599 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:03.531614 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:03.531523 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:05.460140 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:05.460105 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:05.634792 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:05.634754 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:05.940795 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:05.940767 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:06.332543 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:06.332436 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:06.720163 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:06.720121 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:07.226284 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:07.226234 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:08.003708 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:08.003672 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:08.117074 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:08.117046 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:10.222060 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:10.222025 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n2xt8"] Sep 11 14:28:10.224237 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:10.224221 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n2xt8" Sep 11 14:28:10.228304 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:10.228281 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 14:28:10.228821 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:10.228804 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 14:28:10.230649 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:10.230340 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6dd7k\"" Sep 11 14:28:10.286786 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:10.286751 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n2xt8"] Sep 11 14:28:10.295456 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:10.295427 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec43ab8a-2cf9-45cf-bb4c-754612c0027a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-n2xt8\" (UID: \"ec43ab8a-2cf9-45cf-bb4c-754612c0027a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n2xt8" Sep 11 14:28:10.295592 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:10.295463 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc9dr\" (UniqueName: \"kubernetes.io/projected/ec43ab8a-2cf9-45cf-bb4c-754612c0027a-kube-api-access-nc9dr\") pod \"kuadrant-operator-controller-manager-76f8f686bf-n2xt8\" (UID: \"ec43ab8a-2cf9-45cf-bb4c-754612c0027a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n2xt8" Sep 11 14:28:10.396318 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:10.396282 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec43ab8a-2cf9-45cf-bb4c-754612c0027a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-n2xt8\" (UID: \"ec43ab8a-2cf9-45cf-bb4c-754612c0027a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n2xt8" Sep 11 14:28:10.396544 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:10.396325 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc9dr\" (UniqueName: \"kubernetes.io/projected/ec43ab8a-2cf9-45cf-bb4c-754612c0027a-kube-api-access-nc9dr\") pod \"kuadrant-operator-controller-manager-76f8f686bf-n2xt8\" (UID: \"ec43ab8a-2cf9-45cf-bb4c-754612c0027a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n2xt8" Sep 11 14:28:10.396739 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:10.396716 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec43ab8a-2cf9-45cf-bb4c-754612c0027a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-n2xt8\" (UID: \"ec43ab8a-2cf9-45cf-bb4c-754612c0027a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n2xt8" Sep 11 14:28:10.417051 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:10.417024 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc9dr\" (UniqueName: \"kubernetes.io/projected/ec43ab8a-2cf9-45cf-bb4c-754612c0027a-kube-api-access-nc9dr\") pod \"kuadrant-operator-controller-manager-76f8f686bf-n2xt8\" (UID: \"ec43ab8a-2cf9-45cf-bb4c-754612c0027a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n2xt8" Sep 11 14:28:10.487965 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:10.487875 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:10.534171 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:10.534138 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n2xt8" Sep 11 14:28:10.836564 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:10.836486 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n2xt8"] Sep 11 14:28:10.840669 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:28:10.840639 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec43ab8a_2cf9_45cf_bb4c_754612c0027a.slice/crio-25a7f7c64816c177b578a1d419a7464022a0b52659d6808f5b8be17016b4f9b5 WatchSource:0}: Error finding container 25a7f7c64816c177b578a1d419a7464022a0b52659d6808f5b8be17016b4f9b5: Status 404 returned error can't find the container with id 25a7f7c64816c177b578a1d419a7464022a0b52659d6808f5b8be17016b4f9b5 Sep 11 14:28:10.843447 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:10.843429 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:28:11.121114 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:11.121034 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n2xt8" event={"ID":"ec43ab8a-2cf9-45cf-bb4c-754612c0027a","Type":"ContainerStarted","Data":"25a7f7c64816c177b578a1d419a7464022a0b52659d6808f5b8be17016b4f9b5"} Sep 11 14:28:12.500808 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:12.500577 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:13.504343 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:13.504307 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:13.612458 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:13.612393 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:13.697500 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:13.697457 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:16.148977 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:16.148940 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n2xt8" event={"ID":"ec43ab8a-2cf9-45cf-bb4c-754612c0027a","Type":"ContainerStarted","Data":"03ca91987a93296064c689f4035c097844d7a30fb377b5e4fe5b1489bfa25ec0"} Sep 11 14:28:16.149437 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:16.149034 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n2xt8" Sep 11 14:28:16.189218 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:16.189164 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n2xt8" podStartSLOduration=1.934924589 podStartE2EDuration="6.189147175s" totalImagesPullingTime="4.254222586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:28:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:28:16.187638702 +0000 UTC m=+806.801745045" watchObservedRunningTime="2026-09-11 14:28:16.189147175 +0000 UTC m=+806.803253526" Sep 11 14:28:17.514825 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:17.514791 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:17.635859 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:17.635796 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:17.944410 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:17.944374 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:18.047870 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:18.047829 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:18.161850 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:18.161818 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:18.432699 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:18.432668 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:23.370495 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:23.370443 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:23.472463 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:23.472406 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:23.578214 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:23.578180 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:27.156511 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:27.156478 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n2xt8" Sep 11 14:28:29.448792 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.448758 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r"] Sep 11 14:28:29.454081 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.454059 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r" Sep 11 14:28:29.475437 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.475395 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r"] Sep 11 14:28:29.560549 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.560506 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26211b4c-ab87-4142-9748-c796ec99fe80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-d9m6r\" (UID: \"26211b4c-ab87-4142-9748-c796ec99fe80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r" Sep 11 14:28:29.560549 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.560554 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn2ps\" (UniqueName: \"kubernetes.io/projected/26211b4c-ab87-4142-9748-c796ec99fe80-kube-api-access-jn2ps\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-d9m6r\" (UID: \"26211b4c-ab87-4142-9748-c796ec99fe80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r" Sep 11 14:28:29.661357 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.661322 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26211b4c-ab87-4142-9748-c796ec99fe80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-d9m6r\" (UID: \"26211b4c-ab87-4142-9748-c796ec99fe80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r" Sep 11 14:28:29.661530 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.661361 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn2ps\" (UniqueName: \"kubernetes.io/projected/26211b4c-ab87-4142-9748-c796ec99fe80-kube-api-access-jn2ps\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-d9m6r\" (UID: \"26211b4c-ab87-4142-9748-c796ec99fe80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r" Sep 11 14:28:29.661719 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.661686 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26211b4c-ab87-4142-9748-c796ec99fe80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-d9m6r\" (UID: \"26211b4c-ab87-4142-9748-c796ec99fe80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r" Sep 11 14:28:29.672360 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.672335 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn2ps\" (UniqueName: \"kubernetes.io/projected/26211b4c-ab87-4142-9748-c796ec99fe80-kube-api-access-jn2ps\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-d9m6r\" (UID: \"26211b4c-ab87-4142-9748-c796ec99fe80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r" Sep 11 14:28:29.763627 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.763535 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r" Sep 11 14:28:29.770855 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.770830 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n2xt8"] Sep 11 14:28:29.771096 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.771036 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n2xt8" podUID="ec43ab8a-2cf9-45cf-bb4c-754612c0027a" containerName="manager" containerID="cri-o://03ca91987a93296064c689f4035c097844d7a30fb377b5e4fe5b1489bfa25ec0" gracePeriod=2 Sep 11 14:28:29.799524 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.799497 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n2xt8"] Sep 11 14:28:29.823661 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.823631 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r"] Sep 11 14:28:29.831365 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.831329 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r"] Sep 11 14:28:29.873074 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.873048 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk"] Sep 11 14:28:29.873385 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.873369 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec43ab8a-2cf9-45cf-bb4c-754612c0027a" containerName="manager" Sep 11 14:28:29.873385 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.873385 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec43ab8a-2cf9-45cf-bb4c-754612c0027a" containerName="manager" Sep 11 14:28:29.873547 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.873472 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec43ab8a-2cf9-45cf-bb4c-754612c0027a" containerName="manager" Sep 11 14:28:29.876687 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.876665 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk" Sep 11 14:28:29.880169 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:28:29.880134 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-n2xt8\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="ec43ab8a-2cf9-45cf-bb4c-754612c0027a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n2xt8" Sep 11 14:28:29.897746 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.897643 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m"] Sep 11 14:28:29.901506 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.901483 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m" Sep 11 14:28:29.915432 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.915383 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk"] Sep 11 14:28:29.927549 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.927521 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m"] Sep 11 14:28:29.963843 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.963815 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn48s\" (UniqueName: \"kubernetes.io/projected/f50cac37-6e5d-427c-93ab-a388c69eef03-kube-api-access-bn48s\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zggdk\" (UID: \"f50cac37-6e5d-427c-93ab-a388c69eef03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk" Sep 11 14:28:29.963956 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.963935 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f50cac37-6e5d-427c-93ab-a388c69eef03-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zggdk\" (UID: \"f50cac37-6e5d-427c-93ab-a388c69eef03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk" Sep 11 14:28:29.964023 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.963988 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/123a6173-92da-4562-a7ae-366c628357ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-wdl2m\" (UID: \"123a6173-92da-4562-a7ae-366c628357ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m" Sep 11 14:28:29.964089 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:29.964070 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k8z5\" (UniqueName: \"kubernetes.io/projected/123a6173-92da-4562-a7ae-366c628357ea-kube-api-access-6k8z5\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-wdl2m\" (UID: \"123a6173-92da-4562-a7ae-366c628357ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m" Sep 11 14:28:30.025577 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.025555 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n2xt8" Sep 11 14:28:30.064551 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.064516 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn48s\" (UniqueName: \"kubernetes.io/projected/f50cac37-6e5d-427c-93ab-a388c69eef03-kube-api-access-bn48s\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zggdk\" (UID: \"f50cac37-6e5d-427c-93ab-a388c69eef03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk" Sep 11 14:28:30.064708 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.064606 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f50cac37-6e5d-427c-93ab-a388c69eef03-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zggdk\" (UID: \"f50cac37-6e5d-427c-93ab-a388c69eef03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk" Sep 11 14:28:30.064708 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.064656 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/123a6173-92da-4562-a7ae-366c628357ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-wdl2m\" (UID: \"123a6173-92da-4562-a7ae-366c628357ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m" Sep 11 14:28:30.064843 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.064715 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k8z5\" (UniqueName: \"kubernetes.io/projected/123a6173-92da-4562-a7ae-366c628357ea-kube-api-access-6k8z5\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-wdl2m\" (UID: \"123a6173-92da-4562-a7ae-366c628357ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m" Sep 11 14:28:30.065061 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.065030 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/123a6173-92da-4562-a7ae-366c628357ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-wdl2m\" (UID: \"123a6173-92da-4562-a7ae-366c628357ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m" Sep 11 14:28:30.065061 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.065050 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f50cac37-6e5d-427c-93ab-a388c69eef03-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zggdk\" (UID: \"f50cac37-6e5d-427c-93ab-a388c69eef03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk" Sep 11 14:28:30.088316 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.088288 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn48s\" (UniqueName: \"kubernetes.io/projected/f50cac37-6e5d-427c-93ab-a388c69eef03-kube-api-access-bn48s\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zggdk\" (UID: \"f50cac37-6e5d-427c-93ab-a388c69eef03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk" Sep 11 14:28:30.090182 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.090158 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k8z5\" (UniqueName: \"kubernetes.io/projected/123a6173-92da-4562-a7ae-366c628357ea-kube-api-access-6k8z5\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-wdl2m\" (UID: \"123a6173-92da-4562-a7ae-366c628357ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m" Sep 11 14:28:30.158567 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.158532 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:30.165537 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.165509 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nc9dr\" (UniqueName: \"kubernetes.io/projected/ec43ab8a-2cf9-45cf-bb4c-754612c0027a-kube-api-access-nc9dr\") pod \"ec43ab8a-2cf9-45cf-bb4c-754612c0027a\" (UID: \"ec43ab8a-2cf9-45cf-bb4c-754612c0027a\") " Sep 11 14:28:30.165680 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.165567 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec43ab8a-2cf9-45cf-bb4c-754612c0027a-extensions-socket-volume\") pod \"ec43ab8a-2cf9-45cf-bb4c-754612c0027a\" (UID: \"ec43ab8a-2cf9-45cf-bb4c-754612c0027a\") " Sep 11 14:28:30.165984 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.165961 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec43ab8a-2cf9-45cf-bb4c-754612c0027a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ec43ab8a-2cf9-45cf-bb4c-754612c0027a" (UID: "ec43ab8a-2cf9-45cf-bb4c-754612c0027a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:28:30.167690 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.167668 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec43ab8a-2cf9-45cf-bb4c-754612c0027a-kube-api-access-nc9dr" (OuterVolumeSpecName: "kube-api-access-nc9dr") pod "ec43ab8a-2cf9-45cf-bb4c-754612c0027a" (UID: "ec43ab8a-2cf9-45cf-bb4c-754612c0027a"). InnerVolumeSpecName "kube-api-access-nc9dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:28:30.171492 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:28:30.171464 2553 log.go:32] "RunPodSandbox from runtime service failed" err=< Sep 11 14:28:30.171492 ip-10-0-135-17 kubenswrapper[2553]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-d9m6r_kuadrant-system_26211b4c-ab87-4142-9748-c796ec99fe80_0(72d7a1f33f4740123869527768f9fb50bce1a0ab599779d47af0e52ad48d04b0): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-d9m6r to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"72d7a1f33f4740123869527768f9fb50bce1a0ab599779d47af0e52ad48d04b0" Netns:"/var/run/netns/5ec3a350-5cc4-476e-aa46-804b4c0783e5" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-d9m6r;K8S_POD_INFRA_CONTAINER_ID=72d7a1f33f4740123869527768f9fb50bce1a0ab599779d47af0e52ad48d04b0;K8S_POD_UID=26211b4c-ab87-4142-9748-c796ec99fe80" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r/26211b4c-ab87-4142-9748-c796ec99fe80:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r 72d7a1f33f4740123869527768f9fb50bce1a0ab599779d47af0e52ad48d04b0 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:18 [10.132.0.24/23] Sep 11 14:28:30.171492 ip-10-0-135-17 kubenswrapper[2553]: ' Sep 11 14:28:30.171492 ip-10-0-135-17 kubenswrapper[2553]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 11 14:28:30.171492 ip-10-0-135-17 kubenswrapper[2553]: > Sep 11 14:28:30.171680 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:28:30.171516 2553 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Sep 11 14:28:30.171680 ip-10-0-135-17 kubenswrapper[2553]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-d9m6r_kuadrant-system_26211b4c-ab87-4142-9748-c796ec99fe80_0(72d7a1f33f4740123869527768f9fb50bce1a0ab599779d47af0e52ad48d04b0): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-d9m6r to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"72d7a1f33f4740123869527768f9fb50bce1a0ab599779d47af0e52ad48d04b0" Netns:"/var/run/netns/5ec3a350-5cc4-476e-aa46-804b4c0783e5" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-d9m6r;K8S_POD_INFRA_CONTAINER_ID=72d7a1f33f4740123869527768f9fb50bce1a0ab599779d47af0e52ad48d04b0;K8S_POD_UID=26211b4c-ab87-4142-9748-c796ec99fe80" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r/26211b4c-ab87-4142-9748-c796ec99fe80:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r 72d7a1f33f4740123869527768f9fb50bce1a0ab599779d47af0e52ad48d04b0 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:18 [10.132.0.24/23] Sep 11 14:28:30.171680 ip-10-0-135-17 kubenswrapper[2553]: ' Sep 11 14:28:30.171680 ip-10-0-135-17 kubenswrapper[2553]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 11 14:28:30.171680 ip-10-0-135-17 kubenswrapper[2553]: > pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r" Sep 11 14:28:30.189692 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.189670 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk" Sep 11 14:28:30.216505 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.216311 2553 generic.go:356] "Generic (PLEG): container finished" podID="ec43ab8a-2cf9-45cf-bb4c-754612c0027a" containerID="03ca91987a93296064c689f4035c097844d7a30fb377b5e4fe5b1489bfa25ec0" exitCode=0 Sep 11 14:28:30.216505 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.216435 2553 scope.go:117] "RemoveContainer" containerID="03ca91987a93296064c689f4035c097844d7a30fb377b5e4fe5b1489bfa25ec0" Sep 11 14:28:30.216505 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.216443 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m" Sep 11 14:28:30.216505 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.216459 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-n2xt8" Sep 11 14:28:30.216808 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.216717 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r" Sep 11 14:28:30.220138 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:28:30.220106 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-d9m6r\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="26211b4c-ab87-4142-9748-c796ec99fe80" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r" Sep 11 14:28:30.224928 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.224910 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r" Sep 11 14:28:30.226852 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.226832 2553 scope.go:117] "RemoveContainer" containerID="03ca91987a93296064c689f4035c097844d7a30fb377b5e4fe5b1489bfa25ec0" Sep 11 14:28:30.227158 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:28:30.227131 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03ca91987a93296064c689f4035c097844d7a30fb377b5e4fe5b1489bfa25ec0\": container with ID starting with 03ca91987a93296064c689f4035c097844d7a30fb377b5e4fe5b1489bfa25ec0 not found: ID does not exist" containerID="03ca91987a93296064c689f4035c097844d7a30fb377b5e4fe5b1489bfa25ec0" Sep 11 14:28:30.227260 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.227167 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03ca91987a93296064c689f4035c097844d7a30fb377b5e4fe5b1489bfa25ec0"} err="failed to get container status \"03ca91987a93296064c689f4035c097844d7a30fb377b5e4fe5b1489bfa25ec0\": rpc error: code = NotFound desc = could not find container \"03ca91987a93296064c689f4035c097844d7a30fb377b5e4fe5b1489bfa25ec0\": container with ID starting with 03ca91987a93296064c689f4035c097844d7a30fb377b5e4fe5b1489bfa25ec0 not found: ID does not exist" Sep 11 14:28:30.228136 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:28:30.228031 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-d9m6r\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="26211b4c-ab87-4142-9748-c796ec99fe80" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r" Sep 11 14:28:30.233672 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:28:30.233649 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-d9m6r\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="26211b4c-ab87-4142-9748-c796ec99fe80" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r" Sep 11 14:28:30.267281 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.267134 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc9dr\" (UniqueName: \"kubernetes.io/projected/ec43ab8a-2cf9-45cf-bb4c-754612c0027a-kube-api-access-nc9dr\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:28:30.267281 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.267168 2553 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec43ab8a-2cf9-45cf-bb4c-754612c0027a-extensions-socket-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:28:30.354659 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.354622 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:30.368166 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.368144 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jn2ps\" (UniqueName: \"kubernetes.io/projected/26211b4c-ab87-4142-9748-c796ec99fe80-kube-api-access-jn2ps\") pod \"26211b4c-ab87-4142-9748-c796ec99fe80\" (UID: \"26211b4c-ab87-4142-9748-c796ec99fe80\") " Sep 11 14:28:30.368242 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.368195 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26211b4c-ab87-4142-9748-c796ec99fe80-extensions-socket-volume\") pod \"26211b4c-ab87-4142-9748-c796ec99fe80\" (UID: \"26211b4c-ab87-4142-9748-c796ec99fe80\") " Sep 11 14:28:30.368525 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.368496 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26211b4c-ab87-4142-9748-c796ec99fe80-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "26211b4c-ab87-4142-9748-c796ec99fe80" (UID: "26211b4c-ab87-4142-9748-c796ec99fe80"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:28:30.370193 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.370168 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26211b4c-ab87-4142-9748-c796ec99fe80-kube-api-access-jn2ps" (OuterVolumeSpecName: "kube-api-access-jn2ps") pod "26211b4c-ab87-4142-9748-c796ec99fe80" (UID: "26211b4c-ab87-4142-9748-c796ec99fe80"). InnerVolumeSpecName "kube-api-access-jn2ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:28:30.468950 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.468919 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn2ps\" (UniqueName: \"kubernetes.io/projected/26211b4c-ab87-4142-9748-c796ec99fe80-kube-api-access-jn2ps\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:28:30.468950 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.468941 2553 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26211b4c-ab87-4142-9748-c796ec99fe80-extensions-socket-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:28:30.514060 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:28:30.514027 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf50cac37_6e5d_427c_93ab_a388c69eef03.slice/crio-32ad77f5215e8dabe49edd4ca884deecb86603c987c7eb9ed018f04b5ecab05f WatchSource:0}: Error finding container 32ad77f5215e8dabe49edd4ca884deecb86603c987c7eb9ed018f04b5ecab05f: Status 404 returned error can't find the container with id 32ad77f5215e8dabe49edd4ca884deecb86603c987c7eb9ed018f04b5ecab05f Sep 11 14:28:30.514191 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.514170 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk"] Sep 11 14:28:30.587183 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:28:30.587145 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123a6173_92da_4562_a7ae_366c628357ea.slice/crio-c9e574611406c6cc2e0fc32e8f5f93ac08bea2fb85606c2c7ecc755fb81972f1 WatchSource:0}: Error finding container c9e574611406c6cc2e0fc32e8f5f93ac08bea2fb85606c2c7ecc755fb81972f1: Status 404 returned error can't find the container with id c9e574611406c6cc2e0fc32e8f5f93ac08bea2fb85606c2c7ecc755fb81972f1 Sep 11 14:28:30.587585 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:30.587560 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m"] Sep 11 14:28:31.222972 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:31.222936 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk" event={"ID":"f50cac37-6e5d-427c-93ab-a388c69eef03","Type":"ContainerStarted","Data":"01f287df53f98721df7a54934e7416496eb022459feba89d7c2ea46d4148298a"} Sep 11 14:28:31.223187 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:31.222992 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk" event={"ID":"f50cac37-6e5d-427c-93ab-a388c69eef03","Type":"ContainerStarted","Data":"32ad77f5215e8dabe49edd4ca884deecb86603c987c7eb9ed018f04b5ecab05f"} Sep 11 14:28:31.223187 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:31.223048 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk" Sep 11 14:28:31.224786 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:31.224759 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m" event={"ID":"123a6173-92da-4562-a7ae-366c628357ea","Type":"ContainerStarted","Data":"302708df57ed8c2d30618fc2ff8825ff56de00f3173f57faa44f115b579038fe"} Sep 11 14:28:31.224895 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:31.224808 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m" event={"ID":"123a6173-92da-4562-a7ae-366c628357ea","Type":"ContainerStarted","Data":"c9e574611406c6cc2e0fc32e8f5f93ac08bea2fb85606c2c7ecc755fb81972f1"} Sep 11 14:28:31.224895 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:31.224838 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m" Sep 11 14:28:31.225808 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:31.225783 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r" Sep 11 14:28:31.260714 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:31.260658 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk" podStartSLOduration=2.260640517 podStartE2EDuration="2.260640517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:28:31.258383174 +0000 UTC m=+821.872489549" watchObservedRunningTime="2026-09-11 14:28:31.260640517 +0000 UTC m=+821.874746861" Sep 11 14:28:31.261026 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:28:31.260992 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-d9m6r\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="26211b4c-ab87-4142-9748-c796ec99fe80" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r" Sep 11 14:28:31.321488 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:31.321428 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m" podStartSLOduration=2.321398845 podStartE2EDuration="2.321398845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:28:31.321170855 +0000 UTC m=+821.935277221" watchObservedRunningTime="2026-09-11 14:28:31.321398845 +0000 UTC m=+821.935505189" Sep 11 14:28:31.324961 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:28:31.324920 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-d9m6r\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="26211b4c-ab87-4142-9748-c796ec99fe80" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-d9m6r" Sep 11 14:28:31.380285 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:31.380251 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:31.912328 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:31.912296 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26211b4c-ab87-4142-9748-c796ec99fe80" path="/var/lib/kubelet/pods/26211b4c-ab87-4142-9748-c796ec99fe80/volumes" Sep 11 14:28:31.912714 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:31.912566 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec43ab8a-2cf9-45cf-bb4c-754612c0027a" path="/var/lib/kubelet/pods/ec43ab8a-2cf9-45cf-bb4c-754612c0027a/volumes" Sep 11 14:28:35.516284 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:35.516248 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:35.684039 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:35.683991 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:39.410076 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:39.410045 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:39.512368 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:39.512332 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:42.232509 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:42.232477 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m" Sep 11 14:28:42.232865 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:42.232547 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk" Sep 11 14:28:42.463058 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:42.463026 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk"] Sep 11 14:28:42.463250 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:42.463223 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk" podUID="f50cac37-6e5d-427c-93ab-a388c69eef03" containerName="manager" containerID="cri-o://01f287df53f98721df7a54934e7416496eb022459feba89d7c2ea46d4148298a" gracePeriod=10 Sep 11 14:28:42.698904 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:42.698875 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk" Sep 11 14:28:42.779329 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:42.779292 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bn48s\" (UniqueName: \"kubernetes.io/projected/f50cac37-6e5d-427c-93ab-a388c69eef03-kube-api-access-bn48s\") pod \"f50cac37-6e5d-427c-93ab-a388c69eef03\" (UID: \"f50cac37-6e5d-427c-93ab-a388c69eef03\") " Sep 11 14:28:42.779329 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:42.779333 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f50cac37-6e5d-427c-93ab-a388c69eef03-extensions-socket-volume\") pod \"f50cac37-6e5d-427c-93ab-a388c69eef03\" (UID: \"f50cac37-6e5d-427c-93ab-a388c69eef03\") " Sep 11 14:28:42.779777 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:42.779751 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f50cac37-6e5d-427c-93ab-a388c69eef03-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f50cac37-6e5d-427c-93ab-a388c69eef03" (UID: "f50cac37-6e5d-427c-93ab-a388c69eef03"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:28:42.781474 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:42.781448 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f50cac37-6e5d-427c-93ab-a388c69eef03-kube-api-access-bn48s" (OuterVolumeSpecName: "kube-api-access-bn48s") pod "f50cac37-6e5d-427c-93ab-a388c69eef03" (UID: "f50cac37-6e5d-427c-93ab-a388c69eef03"). InnerVolumeSpecName "kube-api-access-bn48s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:28:42.863009 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:42.862930 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp"] Sep 11 14:28:42.863260 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:42.863248 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f50cac37-6e5d-427c-93ab-a388c69eef03" containerName="manager" Sep 11 14:28:42.863314 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:42.863262 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f50cac37-6e5d-427c-93ab-a388c69eef03" containerName="manager" Sep 11 14:28:42.863347 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:42.863340 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f50cac37-6e5d-427c-93ab-a388c69eef03" containerName="manager" Sep 11 14:28:42.865290 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:42.865274 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp" Sep 11 14:28:42.880633 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:42.880612 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn48s\" (UniqueName: \"kubernetes.io/projected/f50cac37-6e5d-427c-93ab-a388c69eef03-kube-api-access-bn48s\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:28:42.880633 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:42.880631 2553 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f50cac37-6e5d-427c-93ab-a388c69eef03-extensions-socket-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:28:42.981882 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:42.981839 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmgxt\" (UniqueName: \"kubernetes.io/projected/5e963db9-4b7b-41dc-b53b-72117b72a478-kube-api-access-tmgxt\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5b9pp\" (UID: \"5e963db9-4b7b-41dc-b53b-72117b72a478\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp" Sep 11 14:28:42.982073 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:42.981953 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e963db9-4b7b-41dc-b53b-72117b72a478-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5b9pp\" (UID: \"5e963db9-4b7b-41dc-b53b-72117b72a478\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp" Sep 11 14:28:42.993244 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:42.993216 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp"] Sep 11 14:28:43.083196 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:43.083161 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmgxt\" (UniqueName: \"kubernetes.io/projected/5e963db9-4b7b-41dc-b53b-72117b72a478-kube-api-access-tmgxt\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5b9pp\" (UID: \"5e963db9-4b7b-41dc-b53b-72117b72a478\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp" Sep 11 14:28:43.083196 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:43.083206 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e963db9-4b7b-41dc-b53b-72117b72a478-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5b9pp\" (UID: \"5e963db9-4b7b-41dc-b53b-72117b72a478\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp" Sep 11 14:28:43.083565 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:43.083548 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e963db9-4b7b-41dc-b53b-72117b72a478-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5b9pp\" (UID: \"5e963db9-4b7b-41dc-b53b-72117b72a478\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp" Sep 11 14:28:43.099051 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:43.099019 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmgxt\" (UniqueName: \"kubernetes.io/projected/5e963db9-4b7b-41dc-b53b-72117b72a478-kube-api-access-tmgxt\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5b9pp\" (UID: \"5e963db9-4b7b-41dc-b53b-72117b72a478\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp" Sep 11 14:28:43.174742 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:43.174708 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp" Sep 11 14:28:43.278291 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:43.278250 2553 generic.go:356] "Generic (PLEG): container finished" podID="f50cac37-6e5d-427c-93ab-a388c69eef03" containerID="01f287df53f98721df7a54934e7416496eb022459feba89d7c2ea46d4148298a" exitCode=0 Sep 11 14:28:43.278680 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:43.278300 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk" Sep 11 14:28:43.278680 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:43.278339 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk" event={"ID":"f50cac37-6e5d-427c-93ab-a388c69eef03","Type":"ContainerDied","Data":"01f287df53f98721df7a54934e7416496eb022459feba89d7c2ea46d4148298a"} Sep 11 14:28:43.278680 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:43.278390 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk" event={"ID":"f50cac37-6e5d-427c-93ab-a388c69eef03","Type":"ContainerDied","Data":"32ad77f5215e8dabe49edd4ca884deecb86603c987c7eb9ed018f04b5ecab05f"} Sep 11 14:28:43.278680 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:43.278423 2553 scope.go:117] "RemoveContainer" containerID="01f287df53f98721df7a54934e7416496eb022459feba89d7c2ea46d4148298a" Sep 11 14:28:43.287380 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:43.287364 2553 scope.go:117] "RemoveContainer" containerID="01f287df53f98721df7a54934e7416496eb022459feba89d7c2ea46d4148298a" Sep 11 14:28:43.287672 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:28:43.287650 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f287df53f98721df7a54934e7416496eb022459feba89d7c2ea46d4148298a\": container with ID starting with 01f287df53f98721df7a54934e7416496eb022459feba89d7c2ea46d4148298a not found: ID does not exist" containerID="01f287df53f98721df7a54934e7416496eb022459feba89d7c2ea46d4148298a" Sep 11 14:28:43.287772 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:43.287675 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f287df53f98721df7a54934e7416496eb022459feba89d7c2ea46d4148298a"} err="failed to get container status \"01f287df53f98721df7a54934e7416496eb022459feba89d7c2ea46d4148298a\": rpc error: code = NotFound desc = could not find container \"01f287df53f98721df7a54934e7416496eb022459feba89d7c2ea46d4148298a\": container with ID starting with 01f287df53f98721df7a54934e7416496eb022459feba89d7c2ea46d4148298a not found: ID does not exist" Sep 11 14:28:43.317792 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:43.317764 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk"] Sep 11 14:28:43.335730 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:43.335706 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zggdk"] Sep 11 14:28:43.463359 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:43.463334 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp"] Sep 11 14:28:43.466124 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:28:43.466098 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e963db9_4b7b_41dc_b53b_72117b72a478.slice/crio-9538d74f56bf16216c4987ad7387daad4ce2cdd2efb3788986bff5ff2fdaea08 WatchSource:0}: Error finding container 9538d74f56bf16216c4987ad7387daad4ce2cdd2efb3788986bff5ff2fdaea08: Status 404 returned error can't find the container with id 9538d74f56bf16216c4987ad7387daad4ce2cdd2efb3788986bff5ff2fdaea08 Sep 11 14:28:43.911747 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:43.911716 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f50cac37-6e5d-427c-93ab-a388c69eef03" path="/var/lib/kubelet/pods/f50cac37-6e5d-427c-93ab-a388c69eef03/volumes" Sep 11 14:28:44.227742 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:44.227639 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:44.284721 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:44.284682 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp" event={"ID":"5e963db9-4b7b-41dc-b53b-72117b72a478","Type":"ContainerStarted","Data":"b6b3866c99ec96434f2b7724151a94296d0141095416a15b5c251f90c799d83a"} Sep 11 14:28:44.285152 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:44.284741 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp" event={"ID":"5e963db9-4b7b-41dc-b53b-72117b72a478","Type":"ContainerStarted","Data":"9538d74f56bf16216c4987ad7387daad4ce2cdd2efb3788986bff5ff2fdaea08"} Sep 11 14:28:44.285152 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:44.284767 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp" Sep 11 14:28:44.330262 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:44.330208 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp" podStartSLOduration=2.330193543 podStartE2EDuration="2.330193543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:28:44.32763468 +0000 UTC m=+834.941741024" watchObservedRunningTime="2026-09-11 14:28:44.330193543 +0000 UTC m=+834.944299883" Sep 11 14:28:44.340317 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:44.340285 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:44.430675 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:44.430635 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:50.760142 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:50.760109 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:50.830853 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:50.830819 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:55.293489 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:55.293456 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp" Sep 11 14:28:55.353590 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:55.353557 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m"] Sep 11 14:28:55.353851 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:55.353814 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m" podUID="123a6173-92da-4562-a7ae-366c628357ea" containerName="manager" containerID="cri-o://302708df57ed8c2d30618fc2ff8825ff56de00f3173f57faa44f115b579038fe" gracePeriod=10 Sep 11 14:28:55.593056 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:55.593031 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m" Sep 11 14:28:55.691005 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:55.690974 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/123a6173-92da-4562-a7ae-366c628357ea-extensions-socket-volume\") pod \"123a6173-92da-4562-a7ae-366c628357ea\" (UID: \"123a6173-92da-4562-a7ae-366c628357ea\") " Sep 11 14:28:55.691179 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:55.691042 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k8z5\" (UniqueName: \"kubernetes.io/projected/123a6173-92da-4562-a7ae-366c628357ea-kube-api-access-6k8z5\") pod \"123a6173-92da-4562-a7ae-366c628357ea\" (UID: \"123a6173-92da-4562-a7ae-366c628357ea\") " Sep 11 14:28:55.691364 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:55.691339 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/123a6173-92da-4562-a7ae-366c628357ea-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "123a6173-92da-4562-a7ae-366c628357ea" (UID: "123a6173-92da-4562-a7ae-366c628357ea"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:28:55.693103 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:55.693084 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123a6173-92da-4562-a7ae-366c628357ea-kube-api-access-6k8z5" (OuterVolumeSpecName: "kube-api-access-6k8z5") pod "123a6173-92da-4562-a7ae-366c628357ea" (UID: "123a6173-92da-4562-a7ae-366c628357ea"). InnerVolumeSpecName "kube-api-access-6k8z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:28:55.754825 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:55.754785 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:55.791672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:55.791646 2553 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/123a6173-92da-4562-a7ae-366c628357ea-extensions-socket-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:28:55.791672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:55.791668 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k8z5\" (UniqueName: \"kubernetes.io/projected/123a6173-92da-4562-a7ae-366c628357ea-kube-api-access-6k8z5\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:28:55.857348 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:55.857258 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:56.341129 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:56.341085 2553 generic.go:356] "Generic (PLEG): container finished" podID="123a6173-92da-4562-a7ae-366c628357ea" containerID="302708df57ed8c2d30618fc2ff8825ff56de00f3173f57faa44f115b579038fe" exitCode=0 Sep 11 14:28:56.341651 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:56.341193 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m" event={"ID":"123a6173-92da-4562-a7ae-366c628357ea","Type":"ContainerDied","Data":"302708df57ed8c2d30618fc2ff8825ff56de00f3173f57faa44f115b579038fe"} Sep 11 14:28:56.341651 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:56.341289 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m" event={"ID":"123a6173-92da-4562-a7ae-366c628357ea","Type":"ContainerDied","Data":"c9e574611406c6cc2e0fc32e8f5f93ac08bea2fb85606c2c7ecc755fb81972f1"} Sep 11 14:28:56.341651 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:56.341328 2553 scope.go:117] "RemoveContainer" containerID="302708df57ed8c2d30618fc2ff8825ff56de00f3173f57faa44f115b579038fe" Sep 11 14:28:56.341651 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:56.341570 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m" Sep 11 14:28:56.354628 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:56.354459 2553 scope.go:117] "RemoveContainer" containerID="302708df57ed8c2d30618fc2ff8825ff56de00f3173f57faa44f115b579038fe" Sep 11 14:28:56.354759 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:28:56.354732 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"302708df57ed8c2d30618fc2ff8825ff56de00f3173f57faa44f115b579038fe\": container with ID starting with 302708df57ed8c2d30618fc2ff8825ff56de00f3173f57faa44f115b579038fe not found: ID does not exist" containerID="302708df57ed8c2d30618fc2ff8825ff56de00f3173f57faa44f115b579038fe" Sep 11 14:28:56.354813 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:56.354782 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"302708df57ed8c2d30618fc2ff8825ff56de00f3173f57faa44f115b579038fe"} err="failed to get container status \"302708df57ed8c2d30618fc2ff8825ff56de00f3173f57faa44f115b579038fe\": rpc error: code = NotFound desc = could not find container \"302708df57ed8c2d30618fc2ff8825ff56de00f3173f57faa44f115b579038fe\": container with ID starting with 302708df57ed8c2d30618fc2ff8825ff56de00f3173f57faa44f115b579038fe not found: ID does not exist" Sep 11 14:28:56.403712 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:56.403675 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m"] Sep 11 14:28:56.450904 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:56.450878 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-wdl2m"] Sep 11 14:28:57.911019 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:57.910988 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="123a6173-92da-4562-a7ae-366c628357ea" path="/var/lib/kubelet/pods/123a6173-92da-4562-a7ae-366c628357ea/volumes" Sep 11 14:28:59.035454 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:59.035399 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:59.187376 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:59.187342 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:59.978171 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:28:59.978137 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:00.090029 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:00.089998 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:00.819461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:00.819404 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:00.957356 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:00.957325 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:05.233654 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:05.233616 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:05.377883 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:05.377832 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:05.469363 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:05.469330 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:11.763032 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:11.762997 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:11.883883 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:11.883841 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:13.192890 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:13.192856 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:13.303927 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:13.303889 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:13.413009 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:13.412972 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:14.089370 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:14.089336 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mknqq"] Sep 11 14:29:14.089676 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:14.089664 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="123a6173-92da-4562-a7ae-366c628357ea" containerName="manager" Sep 11 14:29:14.089726 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:14.089678 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="123a6173-92da-4562-a7ae-366c628357ea" containerName="manager" Sep 11 14:29:14.089764 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:14.089739 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="123a6173-92da-4562-a7ae-366c628357ea" containerName="manager" Sep 11 14:29:14.093880 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:14.093856 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mknqq" Sep 11 14:29:14.097124 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:14.097105 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ndz6h\"" Sep 11 14:29:14.106684 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:14.106659 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mknqq"] Sep 11 14:29:14.136832 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:14.136802 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:14.140816 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:14.140794 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjv4c\" (UniqueName: \"kubernetes.io/projected/976d9c8e-709e-4447-bba7-7c952713740a-kube-api-access-fjv4c\") pod \"authorino-7bcfd8f9cd-mknqq\" (UID: \"976d9c8e-709e-4447-bba7-7c952713740a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mknqq" Sep 11 14:29:14.241496 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:14.241460 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjv4c\" (UniqueName: \"kubernetes.io/projected/976d9c8e-709e-4447-bba7-7c952713740a-kube-api-access-fjv4c\") pod \"authorino-7bcfd8f9cd-mknqq\" (UID: \"976d9c8e-709e-4447-bba7-7c952713740a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mknqq" Sep 11 14:29:14.243534 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:14.243509 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:14.255465 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:14.255445 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjv4c\" (UniqueName: \"kubernetes.io/projected/976d9c8e-709e-4447-bba7-7c952713740a-kube-api-access-fjv4c\") pod \"authorino-7bcfd8f9cd-mknqq\" (UID: \"976d9c8e-709e-4447-bba7-7c952713740a\") " pod="kuadrant-system/authorino-7bcfd8f9cd-mknqq" Sep 11 14:29:14.266766 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:14.266742 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-sbgz8"] Sep 11 14:29:14.269897 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:14.269881 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-sbgz8" Sep 11 14:29:14.279646 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:14.279619 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sbgz8"] Sep 11 14:29:14.342223 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:14.342139 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qzdg\" (UniqueName: \"kubernetes.io/projected/ff234945-c261-4b23-bf65-36aed035c309-kube-api-access-2qzdg\") pod \"authorino-59c974bd48-sbgz8\" (UID: \"ff234945-c261-4b23-bf65-36aed035c309\") " pod="kuadrant-system/authorino-59c974bd48-sbgz8" Sep 11 14:29:14.402407 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:14.402372 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mknqq" Sep 11 14:29:14.442839 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:14.442802 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qzdg\" (UniqueName: \"kubernetes.io/projected/ff234945-c261-4b23-bf65-36aed035c309-kube-api-access-2qzdg\") pod \"authorino-59c974bd48-sbgz8\" (UID: \"ff234945-c261-4b23-bf65-36aed035c309\") " pod="kuadrant-system/authorino-59c974bd48-sbgz8" Sep 11 14:29:14.458707 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:14.458642 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qzdg\" (UniqueName: \"kubernetes.io/projected/ff234945-c261-4b23-bf65-36aed035c309-kube-api-access-2qzdg\") pod \"authorino-59c974bd48-sbgz8\" (UID: \"ff234945-c261-4b23-bf65-36aed035c309\") " pod="kuadrant-system/authorino-59c974bd48-sbgz8" Sep 11 14:29:14.580072 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:14.580037 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-sbgz8" Sep 11 14:29:14.678178 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:14.678152 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mknqq"] Sep 11 14:29:14.679364 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:29:14.679340 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod976d9c8e_709e_4447_bba7_7c952713740a.slice/crio-d5da87d7c5c81cae77b793fbcccaff07e303e05d546c91f7e68d7777c8a60c64 WatchSource:0}: Error finding container d5da87d7c5c81cae77b793fbcccaff07e303e05d546c91f7e68d7777c8a60c64: Status 404 returned error can't find the container with id d5da87d7c5c81cae77b793fbcccaff07e303e05d546c91f7e68d7777c8a60c64 Sep 11 14:29:14.854905 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:14.854879 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sbgz8"] Sep 11 14:29:14.856630 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:29:14.856603 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff234945_c261_4b23_bf65_36aed035c309.slice/crio-b0a28797381e1380f86af49f3b180e2ad11aa4ae033601ff2106d3b01363e642 WatchSource:0}: Error finding container b0a28797381e1380f86af49f3b180e2ad11aa4ae033601ff2106d3b01363e642: Status 404 returned error can't find the container with id b0a28797381e1380f86af49f3b180e2ad11aa4ae033601ff2106d3b01363e642 Sep 11 14:29:15.426138 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:15.426094 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-sbgz8" event={"ID":"ff234945-c261-4b23-bf65-36aed035c309","Type":"ContainerStarted","Data":"b0a28797381e1380f86af49f3b180e2ad11aa4ae033601ff2106d3b01363e642"} Sep 11 14:29:15.427237 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:15.427213 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mknqq" event={"ID":"976d9c8e-709e-4447-bba7-7c952713740a","Type":"ContainerStarted","Data":"d5da87d7c5c81cae77b793fbcccaff07e303e05d546c91f7e68d7777c8a60c64"} Sep 11 14:29:17.438426 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:17.438386 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mknqq" event={"ID":"976d9c8e-709e-4447-bba7-7c952713740a","Type":"ContainerStarted","Data":"a70d3e7402a5a9c19d75c030de48f79b58a3b728bc645ceadae77df1d56ad032"} Sep 11 14:29:17.439963 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:17.439938 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-sbgz8" event={"ID":"ff234945-c261-4b23-bf65-36aed035c309","Type":"ContainerStarted","Data":"38e5c3bc37b8e1d0c5eeec2701587564a7b2eb25e138639e5dd9b0c092c7a807"} Sep 11 14:29:17.459920 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:17.459864 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-mknqq" podStartSLOduration=1.5849383989999999 podStartE2EDuration="3.459847243s" totalImagesPullingTime="1.874908844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:29:17.459315357 +0000 UTC m=+868.073421715" watchObservedRunningTime="2026-09-11 14:29:17.459847243 +0000 UTC m=+868.073953588" Sep 11 14:29:17.487015 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:17.486908 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-sbgz8" podStartSLOduration=1.784625808 podStartE2EDuration="3.486888564s" totalImagesPullingTime="1.702262756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:29:17.48460535 +0000 UTC m=+868.098711696" watchObservedRunningTime="2026-09-11 14:29:17.486888564 +0000 UTC m=+868.100994909" Sep 11 14:29:17.520468 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:17.520430 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mknqq"] Sep 11 14:29:19.448881 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:19.448839 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-mknqq" podUID="976d9c8e-709e-4447-bba7-7c952713740a" containerName="authorino" containerID="cri-o://a70d3e7402a5a9c19d75c030de48f79b58a3b728bc645ceadae77df1d56ad032" gracePeriod=30 Sep 11 14:29:19.686696 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:19.686669 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mknqq" Sep 11 14:29:19.791080 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:19.791002 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjv4c\" (UniqueName: \"kubernetes.io/projected/976d9c8e-709e-4447-bba7-7c952713740a-kube-api-access-fjv4c\") pod \"976d9c8e-709e-4447-bba7-7c952713740a\" (UID: \"976d9c8e-709e-4447-bba7-7c952713740a\") " Sep 11 14:29:19.792983 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:19.792953 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976d9c8e-709e-4447-bba7-7c952713740a-kube-api-access-fjv4c" (OuterVolumeSpecName: "kube-api-access-fjv4c") pod "976d9c8e-709e-4447-bba7-7c952713740a" (UID: "976d9c8e-709e-4447-bba7-7c952713740a"). InnerVolumeSpecName "kube-api-access-fjv4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:29:19.892550 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:19.892515 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjv4c\" (UniqueName: \"kubernetes.io/projected/976d9c8e-709e-4447-bba7-7c952713740a-kube-api-access-fjv4c\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:29:20.454321 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:20.454286 2553 generic.go:356] "Generic (PLEG): container finished" podID="976d9c8e-709e-4447-bba7-7c952713740a" containerID="a70d3e7402a5a9c19d75c030de48f79b58a3b728bc645ceadae77df1d56ad032" exitCode=0 Sep 11 14:29:20.454822 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:20.454336 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-mknqq" Sep 11 14:29:20.454822 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:20.454357 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mknqq" event={"ID":"976d9c8e-709e-4447-bba7-7c952713740a","Type":"ContainerDied","Data":"a70d3e7402a5a9c19d75c030de48f79b58a3b728bc645ceadae77df1d56ad032"} Sep 11 14:29:20.454822 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:20.454388 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-mknqq" event={"ID":"976d9c8e-709e-4447-bba7-7c952713740a","Type":"ContainerDied","Data":"d5da87d7c5c81cae77b793fbcccaff07e303e05d546c91f7e68d7777c8a60c64"} Sep 11 14:29:20.454822 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:20.454404 2553 scope.go:117] "RemoveContainer" containerID="a70d3e7402a5a9c19d75c030de48f79b58a3b728bc645ceadae77df1d56ad032" Sep 11 14:29:20.462464 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:20.462439 2553 scope.go:117] "RemoveContainer" containerID="a70d3e7402a5a9c19d75c030de48f79b58a3b728bc645ceadae77df1d56ad032" Sep 11 14:29:20.462723 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:29:20.462704 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70d3e7402a5a9c19d75c030de48f79b58a3b728bc645ceadae77df1d56ad032\": container with ID starting with a70d3e7402a5a9c19d75c030de48f79b58a3b728bc645ceadae77df1d56ad032 not found: ID does not exist" containerID="a70d3e7402a5a9c19d75c030de48f79b58a3b728bc645ceadae77df1d56ad032" Sep 11 14:29:20.462771 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:20.462733 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70d3e7402a5a9c19d75c030de48f79b58a3b728bc645ceadae77df1d56ad032"} err="failed to get container status \"a70d3e7402a5a9c19d75c030de48f79b58a3b728bc645ceadae77df1d56ad032\": rpc error: code = NotFound desc = could not find container \"a70d3e7402a5a9c19d75c030de48f79b58a3b728bc645ceadae77df1d56ad032\": container with ID starting with a70d3e7402a5a9c19d75c030de48f79b58a3b728bc645ceadae77df1d56ad032 not found: ID does not exist" Sep 11 14:29:20.472222 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:20.472201 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mknqq"] Sep 11 14:29:20.476370 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:20.476341 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-mknqq"] Sep 11 14:29:21.911755 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:21.911724 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="976d9c8e-709e-4447-bba7-7c952713740a" path="/var/lib/kubelet/pods/976d9c8e-709e-4447-bba7-7c952713740a/volumes" Sep 11 14:29:32.996092 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:32.996046 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:33.092766 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:33.092737 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:33.465710 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:33.465671 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:45.817125 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:45.817031 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:46.048616 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.048583 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:46.060105 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.060075 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-x6zh7"] Sep 11 14:29:46.060406 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.060395 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="976d9c8e-709e-4447-bba7-7c952713740a" containerName="authorino" Sep 11 14:29:46.060466 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.060408 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="976d9c8e-709e-4447-bba7-7c952713740a" containerName="authorino" Sep 11 14:29:46.060501 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.060479 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="976d9c8e-709e-4447-bba7-7c952713740a" containerName="authorino" Sep 11 14:29:46.063371 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.063351 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-x6zh7" Sep 11 14:29:46.079223 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.079197 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-x6zh7"] Sep 11 14:29:46.192598 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.192562 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-x6zh7"] Sep 11 14:29:46.192841 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:29:46.192822 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-mcgjk], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-x6zh7" podUID="12048e0a-b519-4479-999b-e06bf923dc52" Sep 11 14:29:46.217243 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.217205 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcgjk\" (UniqueName: \"kubernetes.io/projected/12048e0a-b519-4479-999b-e06bf923dc52-kube-api-access-mcgjk\") pod \"authorino-557f876f69-x6zh7\" (UID: \"12048e0a-b519-4479-999b-e06bf923dc52\") " pod="kuadrant-system/authorino-557f876f69-x6zh7" Sep 11 14:29:46.235526 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.235495 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-q2xln"] Sep 11 14:29:46.238707 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.238686 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-q2xln" Sep 11 14:29:46.242444 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.242396 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 11 14:29:46.253430 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.253388 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-q2xln"] Sep 11 14:29:46.293998 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.293965 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-q2xln"] Sep 11 14:29:46.294250 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:29:46.294230 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-qvwwx tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-qvwwx tls-cert]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-q2xln" podUID="e811e904-c0ed-4e58-b903-9b6aa234a7d7" Sep 11 14:29:46.318643 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.318609 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcgjk\" (UniqueName: \"kubernetes.io/projected/12048e0a-b519-4479-999b-e06bf923dc52-kube-api-access-mcgjk\") pod \"authorino-557f876f69-x6zh7\" (UID: \"12048e0a-b519-4479-999b-e06bf923dc52\") " pod="kuadrant-system/authorino-557f876f69-x6zh7" Sep 11 14:29:46.334694 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.334619 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcgjk\" (UniqueName: \"kubernetes.io/projected/12048e0a-b519-4479-999b-e06bf923dc52-kube-api-access-mcgjk\") pod \"authorino-557f876f69-x6zh7\" (UID: \"12048e0a-b519-4479-999b-e06bf923dc52\") " pod="kuadrant-system/authorino-557f876f69-x6zh7" Sep 11 14:29:46.340879 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.340850 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f9c479969-k6tbh"] Sep 11 14:29:46.344401 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.344383 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f9c479969-k6tbh" Sep 11 14:29:46.361192 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.361169 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f9c479969-k6tbh"] Sep 11 14:29:46.419728 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.419697 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvwwx\" (UniqueName: \"kubernetes.io/projected/e811e904-c0ed-4e58-b903-9b6aa234a7d7-kube-api-access-qvwwx\") pod \"authorino-6d4dfc649f-q2xln\" (UID: \"e811e904-c0ed-4e58-b903-9b6aa234a7d7\") " pod="kuadrant-system/authorino-6d4dfc649f-q2xln" Sep 11 14:29:46.419900 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.419791 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e811e904-c0ed-4e58-b903-9b6aa234a7d7-tls-cert\") pod \"authorino-6d4dfc649f-q2xln\" (UID: \"e811e904-c0ed-4e58-b903-9b6aa234a7d7\") " pod="kuadrant-system/authorino-6d4dfc649f-q2xln" Sep 11 14:29:46.520199 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.520161 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvwwx\" (UniqueName: \"kubernetes.io/projected/e811e904-c0ed-4e58-b903-9b6aa234a7d7-kube-api-access-qvwwx\") pod \"authorino-6d4dfc649f-q2xln\" (UID: \"e811e904-c0ed-4e58-b903-9b6aa234a7d7\") " pod="kuadrant-system/authorino-6d4dfc649f-q2xln" Sep 11 14:29:46.520368 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.520228 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2stbn\" (UniqueName: \"kubernetes.io/projected/a484487a-23b0-4c75-9baf-1be22825a59d-kube-api-access-2stbn\") pod \"authorino-7f9c479969-k6tbh\" (UID: \"a484487a-23b0-4c75-9baf-1be22825a59d\") " pod="kuadrant-system/authorino-7f9c479969-k6tbh" Sep 11 14:29:46.520368 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.520268 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a484487a-23b0-4c75-9baf-1be22825a59d-tls-cert\") pod \"authorino-7f9c479969-k6tbh\" (UID: \"a484487a-23b0-4c75-9baf-1be22825a59d\") " pod="kuadrant-system/authorino-7f9c479969-k6tbh" Sep 11 14:29:46.520368 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.520297 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e811e904-c0ed-4e58-b903-9b6aa234a7d7-tls-cert\") pod \"authorino-6d4dfc649f-q2xln\" (UID: \"e811e904-c0ed-4e58-b903-9b6aa234a7d7\") " pod="kuadrant-system/authorino-6d4dfc649f-q2xln" Sep 11 14:29:46.522741 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.522711 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e811e904-c0ed-4e58-b903-9b6aa234a7d7-tls-cert\") pod \"authorino-6d4dfc649f-q2xln\" (UID: \"e811e904-c0ed-4e58-b903-9b6aa234a7d7\") " pod="kuadrant-system/authorino-6d4dfc649f-q2xln" Sep 11 14:29:46.530353 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.530330 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvwwx\" (UniqueName: \"kubernetes.io/projected/e811e904-c0ed-4e58-b903-9b6aa234a7d7-kube-api-access-qvwwx\") pod \"authorino-6d4dfc649f-q2xln\" (UID: \"e811e904-c0ed-4e58-b903-9b6aa234a7d7\") " pod="kuadrant-system/authorino-6d4dfc649f-q2xln" Sep 11 14:29:46.567280 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.567257 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-q2xln" Sep 11 14:29:46.567437 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.567257 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-x6zh7" Sep 11 14:29:46.574243 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.574217 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-q2xln" Sep 11 14:29:46.576974 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.576957 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-x6zh7" Sep 11 14:29:46.621742 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.621661 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2stbn\" (UniqueName: \"kubernetes.io/projected/a484487a-23b0-4c75-9baf-1be22825a59d-kube-api-access-2stbn\") pod \"authorino-7f9c479969-k6tbh\" (UID: \"a484487a-23b0-4c75-9baf-1be22825a59d\") " pod="kuadrant-system/authorino-7f9c479969-k6tbh" Sep 11 14:29:46.621742 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.621708 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a484487a-23b0-4c75-9baf-1be22825a59d-tls-cert\") pod \"authorino-7f9c479969-k6tbh\" (UID: \"a484487a-23b0-4c75-9baf-1be22825a59d\") " pod="kuadrant-system/authorino-7f9c479969-k6tbh" Sep 11 14:29:46.624051 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.624033 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a484487a-23b0-4c75-9baf-1be22825a59d-tls-cert\") pod \"authorino-7f9c479969-k6tbh\" (UID: \"a484487a-23b0-4c75-9baf-1be22825a59d\") " pod="kuadrant-system/authorino-7f9c479969-k6tbh" Sep 11 14:29:46.631336 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.631317 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2stbn\" (UniqueName: \"kubernetes.io/projected/a484487a-23b0-4c75-9baf-1be22825a59d-kube-api-access-2stbn\") pod \"authorino-7f9c479969-k6tbh\" (UID: \"a484487a-23b0-4c75-9baf-1be22825a59d\") " pod="kuadrant-system/authorino-7f9c479969-k6tbh" Sep 11 14:29:46.653171 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.653145 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f9c479969-k6tbh" Sep 11 14:29:46.721979 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.721954 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcgjk\" (UniqueName: \"kubernetes.io/projected/12048e0a-b519-4479-999b-e06bf923dc52-kube-api-access-mcgjk\") pod \"12048e0a-b519-4479-999b-e06bf923dc52\" (UID: \"12048e0a-b519-4479-999b-e06bf923dc52\") " Sep 11 14:29:46.722103 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.721984 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvwwx\" (UniqueName: \"kubernetes.io/projected/e811e904-c0ed-4e58-b903-9b6aa234a7d7-kube-api-access-qvwwx\") pod \"e811e904-c0ed-4e58-b903-9b6aa234a7d7\" (UID: \"e811e904-c0ed-4e58-b903-9b6aa234a7d7\") " Sep 11 14:29:46.722199 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.722181 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e811e904-c0ed-4e58-b903-9b6aa234a7d7-tls-cert\") pod \"e811e904-c0ed-4e58-b903-9b6aa234a7d7\" (UID: \"e811e904-c0ed-4e58-b903-9b6aa234a7d7\") " Sep 11 14:29:46.724142 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.724100 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12048e0a-b519-4479-999b-e06bf923dc52-kube-api-access-mcgjk" (OuterVolumeSpecName: "kube-api-access-mcgjk") pod "12048e0a-b519-4479-999b-e06bf923dc52" (UID: "12048e0a-b519-4479-999b-e06bf923dc52"). InnerVolumeSpecName "kube-api-access-mcgjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:29:46.724142 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.724107 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e811e904-c0ed-4e58-b903-9b6aa234a7d7-kube-api-access-qvwwx" (OuterVolumeSpecName: "kube-api-access-qvwwx") pod "e811e904-c0ed-4e58-b903-9b6aa234a7d7" (UID: "e811e904-c0ed-4e58-b903-9b6aa234a7d7"). InnerVolumeSpecName "kube-api-access-qvwwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:29:46.724454 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.724391 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e811e904-c0ed-4e58-b903-9b6aa234a7d7-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e811e904-c0ed-4e58-b903-9b6aa234a7d7" (UID: "e811e904-c0ed-4e58-b903-9b6aa234a7d7"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:29:46.823286 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.823230 2553 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e811e904-c0ed-4e58-b903-9b6aa234a7d7-tls-cert\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:29:46.823286 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.823277 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcgjk\" (UniqueName: \"kubernetes.io/projected/12048e0a-b519-4479-999b-e06bf923dc52-kube-api-access-mcgjk\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:29:46.823286 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.823290 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvwwx\" (UniqueName: \"kubernetes.io/projected/e811e904-c0ed-4e58-b903-9b6aa234a7d7-kube-api-access-qvwwx\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:29:46.925466 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:46.925441 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f9c479969-k6tbh"] Sep 11 14:29:46.927490 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:29:46.927462 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda484487a_23b0_4c75_9baf_1be22825a59d.slice/crio-a8211515423b965608afd277fccf9c4215fee9a75814843e3b96dcff3fcd6bb0 WatchSource:0}: Error finding container a8211515423b965608afd277fccf9c4215fee9a75814843e3b96dcff3fcd6bb0: Status 404 returned error can't find the container with id a8211515423b965608afd277fccf9c4215fee9a75814843e3b96dcff3fcd6bb0 Sep 11 14:29:47.572562 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:47.572531 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-x6zh7" Sep 11 14:29:47.572762 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:47.572564 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f9c479969-k6tbh" event={"ID":"a484487a-23b0-4c75-9baf-1be22825a59d","Type":"ContainerStarted","Data":"a8211515423b965608afd277fccf9c4215fee9a75814843e3b96dcff3fcd6bb0"} Sep 11 14:29:47.572762 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:47.572728 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-q2xln" Sep 11 14:29:47.617294 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:47.617258 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-x6zh7"] Sep 11 14:29:47.623272 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:47.623248 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-x6zh7"] Sep 11 14:29:47.659225 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:47.659193 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-q2xln"] Sep 11 14:29:47.664170 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:47.664140 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-q2xln"] Sep 11 14:29:47.911934 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:47.911899 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12048e0a-b519-4479-999b-e06bf923dc52" path="/var/lib/kubelet/pods/12048e0a-b519-4479-999b-e06bf923dc52/volumes" Sep 11 14:29:47.912307 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:47.912194 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e811e904-c0ed-4e58-b903-9b6aa234a7d7" path="/var/lib/kubelet/pods/e811e904-c0ed-4e58-b903-9b6aa234a7d7/volumes" Sep 11 14:29:48.578700 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:48.578658 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f9c479969-k6tbh" event={"ID":"a484487a-23b0-4c75-9baf-1be22825a59d","Type":"ContainerStarted","Data":"61594f7cb514b318abc89d795c6022ef72952afddc3398a153966b0c1f084d98"} Sep 11 14:29:48.601177 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:48.601126 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7f9c479969-k6tbh" podStartSLOduration=1.834616646 podStartE2EDuration="2.601111846s" totalImagesPullingTime="766.4952ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:29:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:29:48.599630048 +0000 UTC m=+899.213736395" watchObservedRunningTime="2026-09-11 14:29:48.601111846 +0000 UTC m=+899.215218190" Sep 11 14:29:48.635246 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:48.635212 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sbgz8"] Sep 11 14:29:48.635675 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:48.635398 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-sbgz8" podUID="ff234945-c261-4b23-bf65-36aed035c309" containerName="authorino" containerID="cri-o://38e5c3bc37b8e1d0c5eeec2701587564a7b2eb25e138639e5dd9b0c092c7a807" gracePeriod=30 Sep 11 14:29:48.875712 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:48.875689 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-sbgz8" Sep 11 14:29:49.044686 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:49.044650 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qzdg\" (UniqueName: \"kubernetes.io/projected/ff234945-c261-4b23-bf65-36aed035c309-kube-api-access-2qzdg\") pod \"ff234945-c261-4b23-bf65-36aed035c309\" (UID: \"ff234945-c261-4b23-bf65-36aed035c309\") " Sep 11 14:29:49.046832 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:49.046794 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff234945-c261-4b23-bf65-36aed035c309-kube-api-access-2qzdg" (OuterVolumeSpecName: "kube-api-access-2qzdg") pod "ff234945-c261-4b23-bf65-36aed035c309" (UID: "ff234945-c261-4b23-bf65-36aed035c309"). InnerVolumeSpecName "kube-api-access-2qzdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:29:49.145263 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:49.145232 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qzdg\" (UniqueName: \"kubernetes.io/projected/ff234945-c261-4b23-bf65-36aed035c309-kube-api-access-2qzdg\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:29:49.585714 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:49.585630 2553 generic.go:356] "Generic (PLEG): container finished" podID="ff234945-c261-4b23-bf65-36aed035c309" containerID="38e5c3bc37b8e1d0c5eeec2701587564a7b2eb25e138639e5dd9b0c092c7a807" exitCode=0 Sep 11 14:29:49.585714 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:49.585675 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-sbgz8" Sep 11 14:29:49.585912 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:49.585713 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-sbgz8" event={"ID":"ff234945-c261-4b23-bf65-36aed035c309","Type":"ContainerDied","Data":"38e5c3bc37b8e1d0c5eeec2701587564a7b2eb25e138639e5dd9b0c092c7a807"} Sep 11 14:29:49.585912 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:49.585756 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-sbgz8" event={"ID":"ff234945-c261-4b23-bf65-36aed035c309","Type":"ContainerDied","Data":"b0a28797381e1380f86af49f3b180e2ad11aa4ae033601ff2106d3b01363e642"} Sep 11 14:29:49.585912 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:49.585775 2553 scope.go:117] "RemoveContainer" containerID="38e5c3bc37b8e1d0c5eeec2701587564a7b2eb25e138639e5dd9b0c092c7a807" Sep 11 14:29:49.594753 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:49.594737 2553 scope.go:117] "RemoveContainer" containerID="38e5c3bc37b8e1d0c5eeec2701587564a7b2eb25e138639e5dd9b0c092c7a807" Sep 11 14:29:49.595055 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:29:49.595037 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38e5c3bc37b8e1d0c5eeec2701587564a7b2eb25e138639e5dd9b0c092c7a807\": container with ID starting with 38e5c3bc37b8e1d0c5eeec2701587564a7b2eb25e138639e5dd9b0c092c7a807 not found: ID does not exist" containerID="38e5c3bc37b8e1d0c5eeec2701587564a7b2eb25e138639e5dd9b0c092c7a807" Sep 11 14:29:49.595115 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:49.595065 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e5c3bc37b8e1d0c5eeec2701587564a7b2eb25e138639e5dd9b0c092c7a807"} err="failed to get container status \"38e5c3bc37b8e1d0c5eeec2701587564a7b2eb25e138639e5dd9b0c092c7a807\": rpc error: code = NotFound desc = could not find container \"38e5c3bc37b8e1d0c5eeec2701587564a7b2eb25e138639e5dd9b0c092c7a807\": container with ID starting with 38e5c3bc37b8e1d0c5eeec2701587564a7b2eb25e138639e5dd9b0c092c7a807 not found: ID does not exist" Sep 11 14:29:49.610449 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:49.610410 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sbgz8"] Sep 11 14:29:49.612683 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:49.612659 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-sbgz8"] Sep 11 14:29:49.879578 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:49.879504 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmqjh_6b2452d9-1730-458c-b380-dbcfd9e5be19/ovn-acl-logging/0.log" Sep 11 14:29:49.880564 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:49.880541 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmqjh_6b2452d9-1730-458c-b380-dbcfd9e5be19/ovn-acl-logging/0.log" Sep 11 14:29:49.912512 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:49.912478 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff234945-c261-4b23-bf65-36aed035c309" path="/var/lib/kubelet/pods/ff234945-c261-4b23-bf65-36aed035c309/volumes" Sep 11 14:29:50.334897 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:50.334860 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:53.545836 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:53.545799 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:53.624617 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:53.624579 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:53.689273 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:53.689237 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:53.756450 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:53.756401 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:53.844338 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:53.844254 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:57.143599 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:57.143564 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:57.243256 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:29:57.243216 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:19.365231 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:19.365192 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:28.509525 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:28.509464 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:28.612443 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:28.612388 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:29.953466 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:29.953429 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:30.052392 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:30.052351 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:30.156342 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:30.156301 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:30.950007 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:30.949973 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:31.074487 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:31.074450 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:31.221889 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:31.221814 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-cvn67"] Sep 11 14:30:31.222149 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:31.222137 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff234945-c261-4b23-bf65-36aed035c309" containerName="authorino" Sep 11 14:30:31.222197 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:31.222151 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff234945-c261-4b23-bf65-36aed035c309" containerName="authorino" Sep 11 14:30:31.222234 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:31.222216 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff234945-c261-4b23-bf65-36aed035c309" containerName="authorino" Sep 11 14:30:31.225191 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:31.225175 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-cvn67" Sep 11 14:30:31.230321 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:31.230299 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 11 14:30:31.230463 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:31.230448 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-c4t8w\"" Sep 11 14:30:31.245553 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:31.245526 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-cvn67"] Sep 11 14:30:31.308402 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:31.308367 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35517420-4a51-4512-810b-834bb79de2f0-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-cvn67\" (UID: \"35517420-4a51-4512-810b-834bb79de2f0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-cvn67" Sep 11 14:30:31.308599 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:31.308439 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vsdh\" (UniqueName: \"kubernetes.io/projected/35517420-4a51-4512-810b-834bb79de2f0-kube-api-access-5vsdh\") pod \"payload-processing-5b54b7fbb7-cvn67\" (UID: \"35517420-4a51-4512-810b-834bb79de2f0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-cvn67" Sep 11 14:30:31.409504 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:31.409462 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35517420-4a51-4512-810b-834bb79de2f0-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-cvn67\" (UID: \"35517420-4a51-4512-810b-834bb79de2f0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-cvn67" Sep 11 14:30:31.409697 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:31.409520 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vsdh\" (UniqueName: \"kubernetes.io/projected/35517420-4a51-4512-810b-834bb79de2f0-kube-api-access-5vsdh\") pod \"payload-processing-5b54b7fbb7-cvn67\" (UID: \"35517420-4a51-4512-810b-834bb79de2f0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-cvn67" Sep 11 14:30:31.410176 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:31.410152 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35517420-4a51-4512-810b-834bb79de2f0-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-cvn67\" (UID: \"35517420-4a51-4512-810b-834bb79de2f0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-cvn67" Sep 11 14:30:31.437636 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:31.437599 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vsdh\" (UniqueName: \"kubernetes.io/projected/35517420-4a51-4512-810b-834bb79de2f0-kube-api-access-5vsdh\") pod \"payload-processing-5b54b7fbb7-cvn67\" (UID: \"35517420-4a51-4512-810b-834bb79de2f0\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-cvn67" Sep 11 14:30:31.535795 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:31.535717 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-cvn67" Sep 11 14:30:31.827764 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:31.827672 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-cvn67"] Sep 11 14:30:31.832364 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:30:31.832303 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35517420_4a51_4512_810b_834bb79de2f0.slice/crio-cf20066321f79212ffbdb50f2e6adf882c4f5f40796b4964e6fb04ea46214ec9 WatchSource:0}: Error finding container cf20066321f79212ffbdb50f2e6adf882c4f5f40796b4964e6fb04ea46214ec9: Status 404 returned error can't find the container with id cf20066321f79212ffbdb50f2e6adf882c4f5f40796b4964e6fb04ea46214ec9 Sep 11 14:30:32.782371 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:32.782334 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-cvn67" event={"ID":"35517420-4a51-4512-810b-834bb79de2f0","Type":"ContainerStarted","Data":"cf20066321f79212ffbdb50f2e6adf882c4f5f40796b4964e6fb04ea46214ec9"} Sep 11 14:30:34.793730 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:34.793694 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-cvn67" event={"ID":"35517420-4a51-4512-810b-834bb79de2f0","Type":"ContainerStarted","Data":"aff8ffe4e2530c2ec2d0088f046d73f25c6669b2a948a46a7ede49693f528950"} Sep 11 14:30:34.794119 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:34.793789 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-cvn67" Sep 11 14:30:40.467492 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:40.467454 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:41.487396 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:41.487360 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:45.800604 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:45.800567 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-cvn67" Sep 11 14:30:45.821847 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:45.821792 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-cvn67" podStartSLOduration=12.341984124 podStartE2EDuration="14.821775205s" totalImagesPullingTime="2.479791081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:30:34.828760598 +0000 UTC m=+945.442866942" watchObservedRunningTime="2026-09-11 14:30:45.821775205 +0000 UTC m=+956.435881548" Sep 11 14:30:49.786943 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:49.786911 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:54.999648 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:54.999614 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9865t"] Sep 11 14:30:55.003793 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.003773 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:30:55.007914 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.007890 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 14:30:55.007914 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.007906 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-n6hc4\"" Sep 11 14:30:55.008117 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.007914 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 11 14:30:55.008117 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.007928 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 14:30:55.025559 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.025532 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9865t"] Sep 11 14:30:55.048663 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.048631 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:55.118464 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.118399 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9865t\" (UID: \"40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:30:55.118464 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.118464 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9865t\" (UID: \"40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:30:55.118696 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.118554 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdfsv\" (UniqueName: \"kubernetes.io/projected/40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6-kube-api-access-pdfsv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9865t\" (UID: \"40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:30:55.118696 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.118588 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9865t\" (UID: \"40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:30:55.118696 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.118648 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9865t\" (UID: \"40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:30:55.118696 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.118686 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9865t\" (UID: \"40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:30:55.219523 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.219482 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9865t\" (UID: \"40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:30:55.219724 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.219571 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9865t\" (UID: \"40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:30:55.219724 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.219596 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9865t\" (UID: \"40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:30:55.219724 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.219629 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdfsv\" (UniqueName: \"kubernetes.io/projected/40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6-kube-api-access-pdfsv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9865t\" (UID: \"40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:30:55.219724 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.219656 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9865t\" (UID: \"40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:30:55.219724 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.219699 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9865t\" (UID: \"40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:30:55.220143 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.220121 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9865t\" (UID: \"40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:30:55.220231 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.220160 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9865t\" (UID: \"40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:30:55.220231 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.220178 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9865t\" (UID: \"40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:30:55.222020 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.221988 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9865t\" (UID: \"40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:30:55.222179 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.222162 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9865t\" (UID: \"40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:30:55.230817 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.230795 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdfsv\" (UniqueName: \"kubernetes.io/projected/40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6-kube-api-access-pdfsv\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9865t\" (UID: \"40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:30:55.314495 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.314400 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:30:55.590626 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.590598 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9865t"] Sep 11 14:30:55.592558 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:30:55.592527 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40dbd1f7_2e0b_4fcd_a2e0_83b618a6b1e6.slice/crio-ec8d3a5d124eb77cf62fa4f6e02123e3efe0773d01763169c2e28efebe07cd11 WatchSource:0}: Error finding container ec8d3a5d124eb77cf62fa4f6e02123e3efe0773d01763169c2e28efebe07cd11: Status 404 returned error can't find the container with id ec8d3a5d124eb77cf62fa4f6e02123e3efe0773d01763169c2e28efebe07cd11 Sep 11 14:30:55.891739 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:55.891699 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" event={"ID":"40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6","Type":"ContainerStarted","Data":"ec8d3a5d124eb77cf62fa4f6e02123e3efe0773d01763169c2e28efebe07cd11"} Sep 11 14:30:56.898040 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:56.898011 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" event={"ID":"40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6","Type":"ContainerStarted","Data":"30c8e47dc221dd0bc44718c11df12a1a8208ed516bdaab7374e9ade38e1ded65"} Sep 11 14:30:56.918180 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:30:56.918118 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" podStartSLOduration=1.685279441 podStartE2EDuration="2.91810008s" totalImagesPullingTime="1.232820639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:30:56.917666043 +0000 UTC m=+967.531772397" watchObservedRunningTime="2026-09-11 14:30:56.91810008 +0000 UTC m=+967.532206427" Sep 11 14:31:00.760378 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:00.760341 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:01.481428 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:01.481381 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:05.314972 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:05.314934 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:31:05.315384 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:05.315011 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:31:05.327461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:05.327434 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:31:05.951435 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:05.951393 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9865t" Sep 11 14:31:21.133120 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.133088 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg"] Sep 11 14:31:21.136727 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.136710 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:21.141965 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.141943 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 11 14:31:21.173886 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.173856 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg"] Sep 11 14:31:21.198359 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.198333 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:21.250314 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.250273 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzggg\" (UID: \"a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:21.250314 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.250319 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzggg\" (UID: \"a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:21.250555 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.250409 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzggg\" (UID: \"a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:21.250555 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.250462 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pndnh\" (UniqueName: \"kubernetes.io/projected/a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5-kube-api-access-pndnh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzggg\" (UID: \"a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:21.250555 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.250482 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzggg\" (UID: \"a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:21.250693 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.250563 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzggg\" (UID: \"a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:21.351533 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.351497 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pndnh\" (UniqueName: \"kubernetes.io/projected/a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5-kube-api-access-pndnh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzggg\" (UID: \"a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:21.351533 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.351538 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzggg\" (UID: \"a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:21.351797 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.351665 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzggg\" (UID: \"a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:21.351797 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.351744 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzggg\" (UID: \"a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:21.351797 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.351766 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzggg\" (UID: \"a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:21.351941 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.351817 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzggg\" (UID: \"a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:21.352185 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.352156 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzggg\" (UID: \"a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:21.352185 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.352175 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzggg\" (UID: \"a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:21.352366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.352217 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzggg\" (UID: \"a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:21.354364 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.354345 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzggg\" (UID: \"a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:21.354554 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.354536 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzggg\" (UID: \"a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:21.362447 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.362408 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pndnh\" (UniqueName: \"kubernetes.io/projected/a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5-kube-api-access-pndnh\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mzggg\" (UID: \"a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:21.446507 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.446467 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:21.731908 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:21.731882 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg"] Sep 11 14:31:21.733624 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:31:21.733596 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07a6e3b_7be3_4bb3_abb8_465dfd2b0ae5.slice/crio-7ee98ce2585b253d4b22ac60ea60eca18926d061951fe7556bc560bb1d463a76 WatchSource:0}: Error finding container 7ee98ce2585b253d4b22ac60ea60eca18926d061951fe7556bc560bb1d463a76: Status 404 returned error can't find the container with id 7ee98ce2585b253d4b22ac60ea60eca18926d061951fe7556bc560bb1d463a76 Sep 11 14:31:22.019175 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:22.019143 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" event={"ID":"a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5","Type":"ContainerStarted","Data":"2379f5794e4e9e0a56fa71fdc159af655f15ce1e84812cc883e487e27ac1ed68"} Sep 11 14:31:22.019298 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:22.019202 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" event={"ID":"a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5","Type":"ContainerStarted","Data":"7ee98ce2585b253d4b22ac60ea60eca18926d061951fe7556bc560bb1d463a76"} Sep 11 14:31:31.048287 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:31.048243 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:31.446609 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:31.446573 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:31.446816 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:31.446734 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:31.459536 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:31.459508 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:31.495763 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:31.495719 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" podStartSLOduration=10.274380547 podStartE2EDuration="10.495704189s" totalImagesPullingTime="221.323642ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:22.05652052 +0000 UTC m=+992.670626864" watchObservedRunningTime="2026-09-11 14:31:31.495704189 +0000 UTC m=+1002.109810532" Sep 11 14:31:32.078884 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:32.078854 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mzggg" Sep 11 14:31:34.791313 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:34.791276 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:42.837730 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:42.837697 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t"] Sep 11 14:31:42.840801 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:42.840781 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:42.844473 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:42.844450 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 11 14:31:42.857932 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:42.857912 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t"] Sep 11 14:31:42.872369 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:42.872337 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:42.945190 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:42.945162 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t\" (UID: \"7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:42.945358 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:42.945239 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t\" (UID: \"7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:42.945358 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:42.945261 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t\" (UID: \"7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:42.945358 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:42.945280 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t\" (UID: \"7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:42.945553 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:42.945370 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-859h2\" (UniqueName: \"kubernetes.io/projected/7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6-kube-api-access-859h2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t\" (UID: \"7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:42.945553 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:42.945465 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t\" (UID: \"7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:43.046395 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:43.046357 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t\" (UID: \"7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:43.046582 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:43.046401 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t\" (UID: \"7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:43.046582 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:43.046456 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t\" (UID: \"7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:43.046582 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:43.046504 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-859h2\" (UniqueName: \"kubernetes.io/projected/7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6-kube-api-access-859h2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t\" (UID: \"7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:43.046582 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:43.046542 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t\" (UID: \"7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:43.046582 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:43.046580 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t\" (UID: \"7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:43.046892 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:43.046866 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t\" (UID: \"7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:43.047086 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:43.047058 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t\" (UID: \"7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:43.047086 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:43.047083 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t\" (UID: \"7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:43.048915 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:43.048886 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t\" (UID: \"7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:43.049058 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:43.049039 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t\" (UID: \"7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:43.057049 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:43.057030 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-859h2\" (UniqueName: \"kubernetes.io/projected/7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6-kube-api-access-859h2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t\" (UID: \"7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:43.150225 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:43.150192 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:43.424288 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:43.424262 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t"] Sep 11 14:31:43.426369 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:31:43.426340 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a17fe65_0779_4a4d_99aa_d9bb1fc58ba6.slice/crio-a286a4bc2a8b75eed6cade8d841f2cbc00be88ffde82d7e997bc6e2e96da0cdc WatchSource:0}: Error finding container a286a4bc2a8b75eed6cade8d841f2cbc00be88ffde82d7e997bc6e2e96da0cdc: Status 404 returned error can't find the container with id a286a4bc2a8b75eed6cade8d841f2cbc00be88ffde82d7e997bc6e2e96da0cdc Sep 11 14:31:44.132735 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:44.132647 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" event={"ID":"7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6","Type":"ContainerStarted","Data":"8c86e8678df9a8b2cf2befd9c65e3dd57e1205cf0f55b36fe7f7a7c86e214c2a"} Sep 11 14:31:44.132735 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:44.132690 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" event={"ID":"7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6","Type":"ContainerStarted","Data":"a286a4bc2a8b75eed6cade8d841f2cbc00be88ffde82d7e997bc6e2e96da0cdc"} Sep 11 14:31:53.151115 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:53.151075 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:53.151115 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:53.151123 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:53.163733 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:53.163711 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:53.185082 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:53.185054 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" Sep 11 14:31:53.192680 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:53.192639 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t" podStartSLOduration=10.994035693 podStartE2EDuration="11.192624561s" totalImagesPullingTime="198.588868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:31:44.15907398 +0000 UTC m=+1014.773180323" watchObservedRunningTime="2026-09-11 14:31:53.192624561 +0000 UTC m=+1023.806730905" Sep 11 14:31:59.397762 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:59.397730 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:31:59.511358 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:31:59.511316 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:10.127387 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:10.127354 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:10.254381 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:10.254340 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:27.994110 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:27.994074 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5645d84896-j49m8"] Sep 11 14:32:27.997537 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:27.997520 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5645d84896-j49m8" Sep 11 14:32:28.000589 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:32:28.000561 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"authorino-5645d84896-j49m8\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="f5ee2a26-df9b-4907-80e6-12c357c2fbb3" pod="kuadrant-system/authorino-5645d84896-j49m8" Sep 11 14:32:28.028172 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:28.028124 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5645d84896-j49m8"] Sep 11 14:32:28.059810 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:28.059776 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f5ee2a26-df9b-4907-80e6-12c357c2fbb3-tls-cert\") pod \"authorino-5645d84896-j49m8\" (UID: \"f5ee2a26-df9b-4907-80e6-12c357c2fbb3\") " pod="kuadrant-system/authorino-5645d84896-j49m8" Sep 11 14:32:28.059977 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:28.059824 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdqgt\" (UniqueName: \"kubernetes.io/projected/f5ee2a26-df9b-4907-80e6-12c357c2fbb3-kube-api-access-sdqgt\") pod \"authorino-5645d84896-j49m8\" (UID: \"f5ee2a26-df9b-4907-80e6-12c357c2fbb3\") " pod="kuadrant-system/authorino-5645d84896-j49m8" Sep 11 14:32:28.161324 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:28.161289 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f5ee2a26-df9b-4907-80e6-12c357c2fbb3-tls-cert\") pod \"authorino-5645d84896-j49m8\" (UID: \"f5ee2a26-df9b-4907-80e6-12c357c2fbb3\") " pod="kuadrant-system/authorino-5645d84896-j49m8" Sep 11 14:32:28.161544 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:28.161342 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdqgt\" (UniqueName: \"kubernetes.io/projected/f5ee2a26-df9b-4907-80e6-12c357c2fbb3-kube-api-access-sdqgt\") pod \"authorino-5645d84896-j49m8\" (UID: \"f5ee2a26-df9b-4907-80e6-12c357c2fbb3\") " pod="kuadrant-system/authorino-5645d84896-j49m8" Sep 11 14:32:28.163739 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:28.163713 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f5ee2a26-df9b-4907-80e6-12c357c2fbb3-tls-cert\") pod \"authorino-5645d84896-j49m8\" (UID: \"f5ee2a26-df9b-4907-80e6-12c357c2fbb3\") " pod="kuadrant-system/authorino-5645d84896-j49m8" Sep 11 14:32:28.173538 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:28.173512 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdqgt\" (UniqueName: \"kubernetes.io/projected/f5ee2a26-df9b-4907-80e6-12c357c2fbb3-kube-api-access-sdqgt\") pod \"authorino-5645d84896-j49m8\" (UID: \"f5ee2a26-df9b-4907-80e6-12c357c2fbb3\") " pod="kuadrant-system/authorino-5645d84896-j49m8" Sep 11 14:32:28.306801 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:28.306720 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5645d84896-j49m8" Sep 11 14:32:28.601562 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:28.601536 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5645d84896-j49m8"] Sep 11 14:32:28.604193 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:32:28.604166 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5ee2a26_df9b_4907_80e6_12c357c2fbb3.slice/crio-7cbe9e5604bdb4c49efd97c48543c407a6b805f54ff9f06275ca4978d80ca442 WatchSource:0}: Error finding container 7cbe9e5604bdb4c49efd97c48543c407a6b805f54ff9f06275ca4978d80ca442: Status 404 returned error can't find the container with id 7cbe9e5604bdb4c49efd97c48543c407a6b805f54ff9f06275ca4978d80ca442 Sep 11 14:32:29.337499 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:29.337472 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5645d84896-j49m8" event={"ID":"f5ee2a26-df9b-4907-80e6-12c357c2fbb3","Type":"ContainerStarted","Data":"7cbe9e5604bdb4c49efd97c48543c407a6b805f54ff9f06275ca4978d80ca442"} Sep 11 14:32:30.346927 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:30.346887 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5645d84896-j49m8" event={"ID":"f5ee2a26-df9b-4907-80e6-12c357c2fbb3","Type":"ContainerStarted","Data":"45ed70eac58b9683ad1d53f0bbf196bc2cac9db8f524b5087088156bce84c6de"} Sep 11 14:32:30.417558 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:30.417507 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5645d84896-j49m8" podStartSLOduration=2.724231933 podStartE2EDuration="3.417491702s" totalImagesPullingTime="693.259769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:32:30.415617105 +0000 UTC m=+1061.029723439" watchObservedRunningTime="2026-09-11 14:32:30.417491702 +0000 UTC m=+1061.031598045" Sep 11 14:32:30.565225 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:30.565187 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f9c479969-k6tbh"] Sep 11 14:32:30.565407 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:30.565384 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7f9c479969-k6tbh" podUID="a484487a-23b0-4c75-9baf-1be22825a59d" containerName="authorino" containerID="cri-o://61594f7cb514b318abc89d795c6022ef72952afddc3398a153966b0c1f084d98" gracePeriod=30 Sep 11 14:32:30.811386 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:30.811356 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f9c479969-k6tbh" Sep 11 14:32:30.887799 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:30.887772 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a484487a-23b0-4c75-9baf-1be22825a59d-tls-cert\") pod \"a484487a-23b0-4c75-9baf-1be22825a59d\" (UID: \"a484487a-23b0-4c75-9baf-1be22825a59d\") " Sep 11 14:32:30.887970 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:30.887816 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2stbn\" (UniqueName: \"kubernetes.io/projected/a484487a-23b0-4c75-9baf-1be22825a59d-kube-api-access-2stbn\") pod \"a484487a-23b0-4c75-9baf-1be22825a59d\" (UID: \"a484487a-23b0-4c75-9baf-1be22825a59d\") " Sep 11 14:32:30.890006 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:30.889973 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a484487a-23b0-4c75-9baf-1be22825a59d-kube-api-access-2stbn" (OuterVolumeSpecName: "kube-api-access-2stbn") pod "a484487a-23b0-4c75-9baf-1be22825a59d" (UID: "a484487a-23b0-4c75-9baf-1be22825a59d"). InnerVolumeSpecName "kube-api-access-2stbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:32:30.903341 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:30.903295 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a484487a-23b0-4c75-9baf-1be22825a59d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a484487a-23b0-4c75-9baf-1be22825a59d" (UID: "a484487a-23b0-4c75-9baf-1be22825a59d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:32:30.989508 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:30.989389 2553 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a484487a-23b0-4c75-9baf-1be22825a59d-tls-cert\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:32:30.989508 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:30.989464 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2stbn\" (UniqueName: \"kubernetes.io/projected/a484487a-23b0-4c75-9baf-1be22825a59d-kube-api-access-2stbn\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:32:31.352694 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:31.352596 2553 generic.go:356] "Generic (PLEG): container finished" podID="a484487a-23b0-4c75-9baf-1be22825a59d" containerID="61594f7cb514b318abc89d795c6022ef72952afddc3398a153966b0c1f084d98" exitCode=0 Sep 11 14:32:31.352694 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:31.352644 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f9c479969-k6tbh" Sep 11 14:32:31.353214 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:31.352686 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f9c479969-k6tbh" event={"ID":"a484487a-23b0-4c75-9baf-1be22825a59d","Type":"ContainerDied","Data":"61594f7cb514b318abc89d795c6022ef72952afddc3398a153966b0c1f084d98"} Sep 11 14:32:31.353214 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:31.352748 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f9c479969-k6tbh" event={"ID":"a484487a-23b0-4c75-9baf-1be22825a59d","Type":"ContainerDied","Data":"a8211515423b965608afd277fccf9c4215fee9a75814843e3b96dcff3fcd6bb0"} Sep 11 14:32:31.353214 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:31.352778 2553 scope.go:117] "RemoveContainer" containerID="61594f7cb514b318abc89d795c6022ef72952afddc3398a153966b0c1f084d98" Sep 11 14:32:31.364488 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:31.364467 2553 scope.go:117] "RemoveContainer" containerID="61594f7cb514b318abc89d795c6022ef72952afddc3398a153966b0c1f084d98" Sep 11 14:32:31.364822 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:32:31.364801 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61594f7cb514b318abc89d795c6022ef72952afddc3398a153966b0c1f084d98\": container with ID starting with 61594f7cb514b318abc89d795c6022ef72952afddc3398a153966b0c1f084d98 not found: ID does not exist" containerID="61594f7cb514b318abc89d795c6022ef72952afddc3398a153966b0c1f084d98" Sep 11 14:32:31.364879 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:31.364831 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61594f7cb514b318abc89d795c6022ef72952afddc3398a153966b0c1f084d98"} err="failed to get container status \"61594f7cb514b318abc89d795c6022ef72952afddc3398a153966b0c1f084d98\": rpc error: code = NotFound desc = could not find container \"61594f7cb514b318abc89d795c6022ef72952afddc3398a153966b0c1f084d98\": container with ID starting with 61594f7cb514b318abc89d795c6022ef72952afddc3398a153966b0c1f084d98 not found: ID does not exist" Sep 11 14:32:31.431046 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:31.431012 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7f9c479969-k6tbh"] Sep 11 14:32:31.450220 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:31.450185 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7f9c479969-k6tbh"] Sep 11 14:32:31.467532 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:31.467504 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:31.675298 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:31.675261 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:31.912191 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:31.912159 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a484487a-23b0-4c75-9baf-1be22825a59d" path="/var/lib/kubelet/pods/a484487a-23b0-4c75-9baf-1be22825a59d/volumes" Sep 11 14:32:33.004921 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:33.004822 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:33.145371 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:33.145338 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:42.678643 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:42.678544 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:50.542768 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:50.542736 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:50.718972 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:50.718938 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:51.025653 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:51.025617 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:51.587935 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:51.587881 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:52.070857 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:52.070817 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:52.166895 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:52.166859 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:52.626572 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:52.626530 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:52.687526 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:52.687495 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7875-8c131b"] Sep 11 14:32:52.687890 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:52.687872 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a484487a-23b0-4c75-9baf-1be22825a59d" containerName="authorino" Sep 11 14:32:52.687890 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:52.687890 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="a484487a-23b0-4c75-9baf-1be22825a59d" containerName="authorino" Sep 11 14:32:52.688055 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:52.687952 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="a484487a-23b0-4c75-9baf-1be22825a59d" containerName="authorino" Sep 11 14:32:52.691773 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:52.691756 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7875-8c131b" Sep 11 14:32:52.695078 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:52.695061 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vdvtq\"" Sep 11 14:32:52.703582 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:52.703562 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7875-8c131b"] Sep 11 14:32:52.781984 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:52.781948 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bjc4\" (UniqueName: \"kubernetes.io/projected/737eef88-3ccd-44c1-bb55-91168fccda59-kube-api-access-5bjc4\") pod \"test-curl-7875-8c131b\" (UID: \"737eef88-3ccd-44c1-bb55-91168fccda59\") " pod="opendatahub/test-curl-7875-8c131b" Sep 11 14:32:52.883205 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:52.883111 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bjc4\" (UniqueName: \"kubernetes.io/projected/737eef88-3ccd-44c1-bb55-91168fccda59-kube-api-access-5bjc4\") pod \"test-curl-7875-8c131b\" (UID: \"737eef88-3ccd-44c1-bb55-91168fccda59\") " pod="opendatahub/test-curl-7875-8c131b" Sep 11 14:32:52.894281 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:52.894253 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bjc4\" (UniqueName: \"kubernetes.io/projected/737eef88-3ccd-44c1-bb55-91168fccda59-kube-api-access-5bjc4\") pod \"test-curl-7875-8c131b\" (UID: \"737eef88-3ccd-44c1-bb55-91168fccda59\") " pod="opendatahub/test-curl-7875-8c131b" Sep 11 14:32:53.001050 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:53.001012 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7875-8c131b" Sep 11 14:32:53.281644 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:53.281613 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7875-8c131b"] Sep 11 14:32:53.284710 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:32:53.284674 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod737eef88_3ccd_44c1_bb55_91168fccda59.slice/crio-a153ae3be1547b3022fce43cd4ebc6116c361c0625822248e58ee04fa0301276 WatchSource:0}: Error finding container a153ae3be1547b3022fce43cd4ebc6116c361c0625822248e58ee04fa0301276: Status 404 returned error can't find the container with id a153ae3be1547b3022fce43cd4ebc6116c361c0625822248e58ee04fa0301276 Sep 11 14:32:53.463472 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:53.463438 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7875-8c131b" event={"ID":"737eef88-3ccd-44c1-bb55-91168fccda59","Type":"ContainerStarted","Data":"a153ae3be1547b3022fce43cd4ebc6116c361c0625822248e58ee04fa0301276"} Sep 11 14:32:54.500429 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:54.500367 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:54.601138 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:54.601101 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:55.473633 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:55.473597 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7875-8c131b" event={"ID":"737eef88-3ccd-44c1-bb55-91168fccda59","Type":"ContainerStarted","Data":"5f9b42806352483d491332aa503dde6428010fbdc0921b3adc2531e6f24928d1"} Sep 11 14:32:55.868563 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:55.868529 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:55.993656 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:55.993570 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:56.089163 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:56.089128 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:56.453099 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:56.453038 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7875-8c131b" podStartSLOduration=2.498784511 podStartE2EDuration="4.453020527s" totalImagesPullingTime="1.954236016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:32:55.518309262 +0000 UTC m=+1086.132415606" watchObservedRunningTime="2026-09-11 14:32:56.453020527 +0000 UTC m=+1087.067126870" Sep 11 14:32:56.453284 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:56.453176 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7875-8c131b"] Sep 11 14:32:56.473075 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:56.473042 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7875-8c131b"] Sep 11 14:32:56.480071 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:32:56.480043 2553 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7875-8c131b" container="test-curl-7875-8c131b" Sep 11 14:32:56.645149 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:56.645108 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7875-93d15a"] Sep 11 14:32:56.645668 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:56.645650 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="737eef88-3ccd-44c1-bb55-91168fccda59" containerName="test-curl-7875-8c131b" Sep 11 14:32:56.645733 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:56.645672 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="737eef88-3ccd-44c1-bb55-91168fccda59" containerName="test-curl-7875-8c131b" Sep 11 14:32:56.645788 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:56.645777 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="737eef88-3ccd-44c1-bb55-91168fccda59" containerName="test-curl-7875-8c131b" Sep 11 14:32:56.649334 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:56.649311 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7875-93d15a" Sep 11 14:32:56.714664 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:56.714589 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7875-93d15a"] Sep 11 14:32:56.717773 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:56.717749 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49lfd\" (UniqueName: \"kubernetes.io/projected/b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7-kube-api-access-49lfd\") pod \"test-curl-7875-93d15a\" (UID: \"b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7\") " pod="opendatahub/test-curl-7875-93d15a" Sep 11 14:32:56.818389 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:56.818355 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49lfd\" (UniqueName: \"kubernetes.io/projected/b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7-kube-api-access-49lfd\") pod \"test-curl-7875-93d15a\" (UID: \"b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7\") " pod="opendatahub/test-curl-7875-93d15a" Sep 11 14:32:56.832710 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:56.832678 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49lfd\" (UniqueName: \"kubernetes.io/projected/b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7-kube-api-access-49lfd\") pod \"test-curl-7875-93d15a\" (UID: \"b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7\") " pod="opendatahub/test-curl-7875-93d15a" Sep 11 14:32:56.958727 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:56.958686 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7875-93d15a" Sep 11 14:32:57.215368 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:57.215340 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:57.267856 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:57.267835 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7875-93d15a"] Sep 11 14:32:57.270187 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:32:57.270159 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8a570c7_0aa6_4c1c_8a65_4b678aa64cf7.slice/crio-f504c4f1fd050485a41f48963332950a6a6b73c74e755f9d5220420e1255a03d WatchSource:0}: Error finding container f504c4f1fd050485a41f48963332950a6a6b73c74e755f9d5220420e1255a03d: Status 404 returned error can't find the container with id f504c4f1fd050485a41f48963332950a6a6b73c74e755f9d5220420e1255a03d Sep 11 14:32:57.380516 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:57.380477 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:57.485610 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:57.485568 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7875-93d15a" event={"ID":"b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7","Type":"ContainerStarted","Data":"18307387ecf7625a6fbcc4b1f22758e8ca1c6a4ee5e8282a96948c66c3fb5b4c"} Sep 11 14:32:57.485804 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:57.485619 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7875-93d15a" event={"ID":"b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7","Type":"ContainerStarted","Data":"f504c4f1fd050485a41f48963332950a6a6b73c74e755f9d5220420e1255a03d"} Sep 11 14:32:57.485804 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:57.485635 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7875-8c131b" podUID="737eef88-3ccd-44c1-bb55-91168fccda59" containerName="test-curl-7875-8c131b" containerID="cri-o://5f9b42806352483d491332aa503dde6428010fbdc0921b3adc2531e6f24928d1" gracePeriod=2 Sep 11 14:32:57.534486 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:32:57.534374 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7875-8c131b\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="737eef88-3ccd-44c1-bb55-91168fccda59" pod="opendatahub/test-curl-7875-8c131b" Sep 11 14:32:57.586024 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:57.585970 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7875-93d15a" podStartSLOduration=1.5859543299999999 podStartE2EDuration="1.58595433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:32:57.58458779 +0000 UTC m=+1088.198694133" watchObservedRunningTime="2026-09-11 14:32:57.58595433 +0000 UTC m=+1088.200060674" Sep 11 14:32:57.590683 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:32:57.590654 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7875-8c131b\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="737eef88-3ccd-44c1-bb55-91168fccda59" pod="opendatahub/test-curl-7875-8c131b" Sep 11 14:32:57.690171 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:57.690142 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv"] Sep 11 14:32:57.693586 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:57.693570 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv" Sep 11 14:32:57.705680 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:32:57.705649 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7875-8c131b\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="737eef88-3ccd-44c1-bb55-91168fccda59" pod="opendatahub/test-curl-7875-8c131b" Sep 11 14:32:57.706380 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:57.706359 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d4444b\"" Sep 11 14:32:57.726942 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:57.726916 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbaed836-34dd-4d8c-82c6-c6c23225c431-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv\" (UID: \"fbaed836-34dd-4d8c-82c6-c6c23225c431\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv" Sep 11 14:32:57.727089 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:57.727000 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlbxc\" (UniqueName: \"kubernetes.io/projected/fbaed836-34dd-4d8c-82c6-c6c23225c431-kube-api-access-nlbxc\") pod \"payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv\" (UID: \"fbaed836-34dd-4d8c-82c6-c6c23225c431\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv" Sep 11 14:32:57.729952 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:57.729916 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv"] Sep 11 14:32:57.736340 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:57.736318 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d4444b-dockercfg-jnhvv\"" Sep 11 14:32:57.828034 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:57.827945 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbaed836-34dd-4d8c-82c6-c6c23225c431-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv\" (UID: \"fbaed836-34dd-4d8c-82c6-c6c23225c431\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv" Sep 11 14:32:57.828034 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:57.828001 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlbxc\" (UniqueName: \"kubernetes.io/projected/fbaed836-34dd-4d8c-82c6-c6c23225c431-kube-api-access-nlbxc\") pod \"payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv\" (UID: \"fbaed836-34dd-4d8c-82c6-c6c23225c431\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv" Sep 11 14:32:57.828738 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:57.828712 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbaed836-34dd-4d8c-82c6-c6c23225c431-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv\" (UID: \"fbaed836-34dd-4d8c-82c6-c6c23225c431\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv" Sep 11 14:32:57.870256 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:57.870228 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlbxc\" (UniqueName: \"kubernetes.io/projected/fbaed836-34dd-4d8c-82c6-c6c23225c431-kube-api-access-nlbxc\") pod \"payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv\" (UID: \"fbaed836-34dd-4d8c-82c6-c6c23225c431\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv" Sep 11 14:32:57.929724 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:57.929694 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7"] Sep 11 14:32:57.933210 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:57.933194 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7" Sep 11 14:32:57.972582 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:57.972555 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7"] Sep 11 14:32:58.003949 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:58.003926 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv" Sep 11 14:32:58.030257 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:58.030226 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12510bdd-4c7e-45b1-81bb-0289660fc613-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7\" (UID: \"12510bdd-4c7e-45b1-81bb-0289660fc613\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7" Sep 11 14:32:58.030385 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:58.030280 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhxjb\" (UniqueName: \"kubernetes.io/projected/12510bdd-4c7e-45b1-81bb-0289660fc613-kube-api-access-nhxjb\") pod \"payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7\" (UID: \"12510bdd-4c7e-45b1-81bb-0289660fc613\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7" Sep 11 14:32:58.130809 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:58.130726 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12510bdd-4c7e-45b1-81bb-0289660fc613-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7\" (UID: \"12510bdd-4c7e-45b1-81bb-0289660fc613\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7" Sep 11 14:32:58.130809 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:58.130771 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhxjb\" (UniqueName: \"kubernetes.io/projected/12510bdd-4c7e-45b1-81bb-0289660fc613-kube-api-access-nhxjb\") pod \"payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7\" (UID: \"12510bdd-4c7e-45b1-81bb-0289660fc613\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7" Sep 11 14:32:58.131551 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:58.131531 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12510bdd-4c7e-45b1-81bb-0289660fc613-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7\" (UID: \"12510bdd-4c7e-45b1-81bb-0289660fc613\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7" Sep 11 14:32:58.155148 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:58.155122 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhxjb\" (UniqueName: \"kubernetes.io/projected/12510bdd-4c7e-45b1-81bb-0289660fc613-kube-api-access-nhxjb\") pod \"payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7\" (UID: \"12510bdd-4c7e-45b1-81bb-0289660fc613\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7" Sep 11 14:32:58.243346 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:58.243311 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7" Sep 11 14:32:58.316956 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:58.316928 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv"] Sep 11 14:32:58.318802 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:32:58.318780 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbaed836_34dd_4d8c_82c6_c6c23225c431.slice/crio-50801c4fec7d360619e8c5040890dafdc5b2ac8bea39abf0984132e99d8bad6d WatchSource:0}: Error finding container 50801c4fec7d360619e8c5040890dafdc5b2ac8bea39abf0984132e99d8bad6d: Status 404 returned error can't find the container with id 50801c4fec7d360619e8c5040890dafdc5b2ac8bea39abf0984132e99d8bad6d Sep 11 14:32:58.491370 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:58.491336 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv" event={"ID":"fbaed836-34dd-4d8c-82c6-c6c23225c431","Type":"ContainerStarted","Data":"dd15c3f4e06dabd2f07e4f057390ac36c733537578ea4f3bc2283313d1721dd6"} Sep 11 14:32:58.491575 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:58.491385 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv" event={"ID":"fbaed836-34dd-4d8c-82c6-c6c23225c431","Type":"ContainerStarted","Data":"50801c4fec7d360619e8c5040890dafdc5b2ac8bea39abf0984132e99d8bad6d"} Sep 11 14:32:58.550879 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:58.550817 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv" podStartSLOduration=1.5508000530000001 podStartE2EDuration="1.550800053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:32:58.543741994 +0000 UTC m=+1089.157848337" watchObservedRunningTime="2026-09-11 14:32:58.550800053 +0000 UTC m=+1089.164906397" Sep 11 14:32:58.558094 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:58.558070 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7"] Sep 11 14:32:58.560422 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:32:58.560390 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12510bdd_4c7e_45b1_81bb_0289660fc613.slice/crio-40d4e856a4297754336bded9b27eedc2bdda3685cc1631f3af30d18b5c0d6416 WatchSource:0}: Error finding container 40d4e856a4297754336bded9b27eedc2bdda3685cc1631f3af30d18b5c0d6416: Status 404 returned error can't find the container with id 40d4e856a4297754336bded9b27eedc2bdda3685cc1631f3af30d18b5c0d6416 Sep 11 14:32:58.848031 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:58.847921 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7875-93d15a"] Sep 11 14:32:58.863515 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:58.863478 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7875-93d15a"] Sep 11 14:32:59.444501 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:59.444448 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:59.499654 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:59.499619 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7" event={"ID":"12510bdd-4c7e-45b1-81bb-0289660fc613","Type":"ContainerStarted","Data":"f58f605e596752848043fa5e822fcd1d603c6b5fbe4b397b4e7717722a89179f"} Sep 11 14:32:59.499775 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:59.499688 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7" event={"ID":"12510bdd-4c7e-45b1-81bb-0289660fc613","Type":"ContainerStarted","Data":"40d4e856a4297754336bded9b27eedc2bdda3685cc1631f3af30d18b5c0d6416"} Sep 11 14:32:59.499775 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:59.499719 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7875-93d15a" podUID="b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7" containerName="test-curl-7875-93d15a" containerID="cri-o://18307387ecf7625a6fbcc4b1f22758e8ca1c6a4ee5e8282a96948c66c3fb5b4c" gracePeriod=2 Sep 11 14:32:59.500273 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:59.500220 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv" Sep 11 14:32:59.539012 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:59.538955 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7" podStartSLOduration=2.538932961 podStartE2EDuration="2.538932961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:32:59.537143951 +0000 UTC m=+1090.151250297" watchObservedRunningTime="2026-09-11 14:32:59.538932961 +0000 UTC m=+1090.153039313" Sep 11 14:32:59.539673 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:32:59.539636 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7875-93d15a\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7" pod="opendatahub/test-curl-7875-93d15a" Sep 11 14:32:59.541919 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:32:59.541888 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7875-93d15a\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7" pod="opendatahub/test-curl-7875-93d15a" Sep 11 14:32:59.659149 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:59.659116 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7875-8c131b" Sep 11 14:32:59.661629 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:32:59.661600 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7875-93d15a\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7" pod="opendatahub/test-curl-7875-93d15a" Sep 11 14:32:59.744572 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:59.744486 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bjc4\" (UniqueName: \"kubernetes.io/projected/737eef88-3ccd-44c1-bb55-91168fccda59-kube-api-access-5bjc4\") pod \"737eef88-3ccd-44c1-bb55-91168fccda59\" (UID: \"737eef88-3ccd-44c1-bb55-91168fccda59\") " Sep 11 14:32:59.747214 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:59.747180 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/737eef88-3ccd-44c1-bb55-91168fccda59-kube-api-access-5bjc4" (OuterVolumeSpecName: "kube-api-access-5bjc4") pod "737eef88-3ccd-44c1-bb55-91168fccda59" (UID: "737eef88-3ccd-44c1-bb55-91168fccda59"). InnerVolumeSpecName "kube-api-access-5bjc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:32:59.845491 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:59.845455 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bjc4\" (UniqueName: \"kubernetes.io/projected/737eef88-3ccd-44c1-bb55-91168fccda59-kube-api-access-5bjc4\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:32:59.887474 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:59.887439 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:32:59.912878 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:32:59.912847 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="737eef88-3ccd-44c1-bb55-91168fccda59" path="/var/lib/kubelet/pods/737eef88-3ccd-44c1-bb55-91168fccda59/volumes" Sep 11 14:32:59.918313 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:32:59.918282 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7875-93d15a\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7" pod="opendatahub/test-curl-7875-93d15a" Sep 11 14:33:00.283650 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:00.283608 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:00.511477 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:00.511440 2553 generic.go:356] "Generic (PLEG): container finished" podID="737eef88-3ccd-44c1-bb55-91168fccda59" containerID="5f9b42806352483d491332aa503dde6428010fbdc0921b3adc2531e6f24928d1" exitCode=137 Sep 11 14:33:00.511955 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:00.511497 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7875-8c131b" Sep 11 14:33:00.511955 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:00.511541 2553 scope.go:117] "RemoveContainer" containerID="5f9b42806352483d491332aa503dde6428010fbdc0921b3adc2531e6f24928d1" Sep 11 14:33:00.511955 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:00.511897 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7" Sep 11 14:33:00.519748 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:00.519730 2553 scope.go:117] "RemoveContainer" containerID="5f9b42806352483d491332aa503dde6428010fbdc0921b3adc2531e6f24928d1" Sep 11 14:33:00.520022 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:33:00.519995 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f9b42806352483d491332aa503dde6428010fbdc0921b3adc2531e6f24928d1\": container with ID starting with 5f9b42806352483d491332aa503dde6428010fbdc0921b3adc2531e6f24928d1 not found: ID does not exist" containerID="5f9b42806352483d491332aa503dde6428010fbdc0921b3adc2531e6f24928d1" Sep 11 14:33:00.520088 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:00.520033 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f9b42806352483d491332aa503dde6428010fbdc0921b3adc2531e6f24928d1"} err="failed to get container status \"5f9b42806352483d491332aa503dde6428010fbdc0921b3adc2531e6f24928d1\": rpc error: code = NotFound desc = could not find container \"5f9b42806352483d491332aa503dde6428010fbdc0921b3adc2531e6f24928d1\": container with ID starting with 5f9b42806352483d491332aa503dde6428010fbdc0921b3adc2531e6f24928d1 not found: ID does not exist" Sep 11 14:33:01.688713 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:01.688690 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7875-93d15a" Sep 11 14:33:01.764987 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:01.764884 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49lfd\" (UniqueName: \"kubernetes.io/projected/b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7-kube-api-access-49lfd\") pod \"b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7\" (UID: \"b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7\") " Sep 11 14:33:01.766906 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:01.766877 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7-kube-api-access-49lfd" (OuterVolumeSpecName: "kube-api-access-49lfd") pod "b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7" (UID: "b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7"). InnerVolumeSpecName "kube-api-access-49lfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:33:01.866316 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:01.866268 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49lfd\" (UniqueName: \"kubernetes.io/projected/b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7-kube-api-access-49lfd\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:33:01.912268 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:01.912234 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7" path="/var/lib/kubelet/pods/b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7/volumes" Sep 11 14:33:02.524036 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:02.523993 2553 generic.go:356] "Generic (PLEG): container finished" podID="b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7" containerID="18307387ecf7625a6fbcc4b1f22758e8ca1c6a4ee5e8282a96948c66c3fb5b4c" exitCode=137 Sep 11 14:33:02.524241 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:02.524041 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7875-93d15a" Sep 11 14:33:02.524241 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:02.524097 2553 scope.go:117] "RemoveContainer" containerID="18307387ecf7625a6fbcc4b1f22758e8ca1c6a4ee5e8282a96948c66c3fb5b4c" Sep 11 14:33:02.533718 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:02.533694 2553 scope.go:117] "RemoveContainer" containerID="18307387ecf7625a6fbcc4b1f22758e8ca1c6a4ee5e8282a96948c66c3fb5b4c" Sep 11 14:33:02.534097 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:33:02.534071 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18307387ecf7625a6fbcc4b1f22758e8ca1c6a4ee5e8282a96948c66c3fb5b4c\": container with ID starting with 18307387ecf7625a6fbcc4b1f22758e8ca1c6a4ee5e8282a96948c66c3fb5b4c not found: ID does not exist" containerID="18307387ecf7625a6fbcc4b1f22758e8ca1c6a4ee5e8282a96948c66c3fb5b4c" Sep 11 14:33:02.534198 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:02.534103 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18307387ecf7625a6fbcc4b1f22758e8ca1c6a4ee5e8282a96948c66c3fb5b4c"} err="failed to get container status \"18307387ecf7625a6fbcc4b1f22758e8ca1c6a4ee5e8282a96948c66c3fb5b4c\": rpc error: code = NotFound desc = could not find container \"18307387ecf7625a6fbcc4b1f22758e8ca1c6a4ee5e8282a96948c66c3fb5b4c\": container with ID starting with 18307387ecf7625a6fbcc4b1f22758e8ca1c6a4ee5e8282a96948c66c3fb5b4c not found: ID does not exist" Sep 11 14:33:02.706252 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:02.706215 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:03.376237 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:03.376199 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:03.564160 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:03.564124 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:03.699006 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:03.698973 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:03.928970 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:03.928933 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:04.130879 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:04.130778 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:04.945616 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:04.945585 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:05.277768 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:05.277686 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:06.297258 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:06.297222 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:07.918618 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:07.918586 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:08.030110 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:08.030074 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:08.322507 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:08.322395 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:08.518395 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:08.518356 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:09.322208 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:09.322169 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:09.656983 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:09.656932 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:09.794453 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:09.794406 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:09.919276 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:09.919183 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:10.161646 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.161613 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd"] Sep 11 14:33:10.161994 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.161982 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7" containerName="test-curl-7875-93d15a" Sep 11 14:33:10.162042 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.161996 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7" containerName="test-curl-7875-93d15a" Sep 11 14:33:10.162094 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.162084 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8a570c7-0aa6-4c1c-8a65-4b678aa64cf7" containerName="test-curl-7875-93d15a" Sep 11 14:33:10.166240 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.166224 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd" Sep 11 14:33:10.173858 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.173831 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-c322a2\"" Sep 11 14:33:10.194264 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.194236 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-c322a2-dockercfg-9qjtc\"" Sep 11 14:33:10.227343 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.227319 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd"] Sep 11 14:33:10.251894 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.251862 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c7493d-982c-44d9-b847-b930de23e261-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd\" (UID: \"f1c7493d-982c-44d9-b847-b930de23e261\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd" Sep 11 14:33:10.252043 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.251914 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjbtm\" (UniqueName: \"kubernetes.io/projected/f1c7493d-982c-44d9-b847-b930de23e261-kube-api-access-gjbtm\") pod \"payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd\" (UID: \"f1c7493d-982c-44d9-b847-b930de23e261\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd" Sep 11 14:33:10.273462 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.273426 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk"] Sep 11 14:33:10.276859 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.276837 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk" Sep 11 14:33:10.315074 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.315043 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk"] Sep 11 14:33:10.352992 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.352956 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c7493d-982c-44d9-b847-b930de23e261-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd\" (UID: \"f1c7493d-982c-44d9-b847-b930de23e261\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd" Sep 11 14:33:10.353462 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.353029 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjbtm\" (UniqueName: \"kubernetes.io/projected/f1c7493d-982c-44d9-b847-b930de23e261-kube-api-access-gjbtm\") pod \"payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd\" (UID: \"f1c7493d-982c-44d9-b847-b930de23e261\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd" Sep 11 14:33:10.353462 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.353061 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee479f4-3ca8-485a-ba93-b6cfa8808775-plugins-config-volume\") pod \"payload-processing-e2e-autores-c322a2-57d754d867-bgffk\" (UID: \"1ee479f4-3ca8-485a-ba93-b6cfa8808775\") " pod="openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk" Sep 11 14:33:10.353462 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.353091 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv4tx\" (UniqueName: \"kubernetes.io/projected/1ee479f4-3ca8-485a-ba93-b6cfa8808775-kube-api-access-sv4tx\") pod \"payload-processing-e2e-autores-c322a2-57d754d867-bgffk\" (UID: \"1ee479f4-3ca8-485a-ba93-b6cfa8808775\") " pod="openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk" Sep 11 14:33:10.353943 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.353919 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c7493d-982c-44d9-b847-b930de23e261-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd\" (UID: \"f1c7493d-982c-44d9-b847-b930de23e261\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd" Sep 11 14:33:10.396304 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.396274 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjbtm\" (UniqueName: \"kubernetes.io/projected/f1c7493d-982c-44d9-b847-b930de23e261-kube-api-access-gjbtm\") pod \"payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd\" (UID: \"f1c7493d-982c-44d9-b847-b930de23e261\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd" Sep 11 14:33:10.424091 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.424008 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:10.453940 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.453899 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee479f4-3ca8-485a-ba93-b6cfa8808775-plugins-config-volume\") pod \"payload-processing-e2e-autores-c322a2-57d754d867-bgffk\" (UID: \"1ee479f4-3ca8-485a-ba93-b6cfa8808775\") " pod="openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk" Sep 11 14:33:10.454090 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.453947 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv4tx\" (UniqueName: \"kubernetes.io/projected/1ee479f4-3ca8-485a-ba93-b6cfa8808775-kube-api-access-sv4tx\") pod \"payload-processing-e2e-autores-c322a2-57d754d867-bgffk\" (UID: \"1ee479f4-3ca8-485a-ba93-b6cfa8808775\") " pod="openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk" Sep 11 14:33:10.454629 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.454609 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee479f4-3ca8-485a-ba93-b6cfa8808775-plugins-config-volume\") pod \"payload-processing-e2e-autores-c322a2-57d754d867-bgffk\" (UID: \"1ee479f4-3ca8-485a-ba93-b6cfa8808775\") " pod="openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk" Sep 11 14:33:10.475866 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.475837 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd" Sep 11 14:33:10.483817 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.483789 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv4tx\" (UniqueName: \"kubernetes.io/projected/1ee479f4-3ca8-485a-ba93-b6cfa8808775-kube-api-access-sv4tx\") pod \"payload-processing-e2e-autores-c322a2-57d754d867-bgffk\" (UID: \"1ee479f4-3ca8-485a-ba93-b6cfa8808775\") " pod="openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk" Sep 11 14:33:10.514391 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.514363 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv" Sep 11 14:33:10.588459 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.588117 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk" Sep 11 14:33:10.772314 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.772224 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:10.808485 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.808444 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd"] Sep 11 14:33:10.812579 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:33:10.812548 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c7493d_982c_44d9_b847_b930de23e261.slice/crio-5934553c981eb0dbaaaf8360aa645659dc01549d6896722c4cc0b940c5e7772e WatchSource:0}: Error finding container 5934553c981eb0dbaaaf8360aa645659dc01549d6896722c4cc0b940c5e7772e: Status 404 returned error can't find the container with id 5934553c981eb0dbaaaf8360aa645659dc01549d6896722c4cc0b940c5e7772e Sep 11 14:33:10.940536 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:33:10.940498 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee479f4_3ca8_485a_ba93_b6cfa8808775.slice/crio-68add7603767fe2f1ffc0f648739b82ae69830ad224e3c9bd92a413d2a1959d6 WatchSource:0}: Error finding container 68add7603767fe2f1ffc0f648739b82ae69830ad224e3c9bd92a413d2a1959d6: Status 404 returned error can't find the container with id 68add7603767fe2f1ffc0f648739b82ae69830ad224e3c9bd92a413d2a1959d6 Sep 11 14:33:10.942647 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:10.942625 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk"] Sep 11 14:33:11.181679 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:11.181639 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:11.311089 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:11.311040 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:11.519982 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:11.519899 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7" Sep 11 14:33:11.584167 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:11.584128 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk" event={"ID":"1ee479f4-3ca8-485a-ba93-b6cfa8808775","Type":"ContainerStarted","Data":"268f0354a82f80140536cdfc881a0310a88400d05987e22f3db785b653c4f151"} Sep 11 14:33:11.584350 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:11.584190 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk" event={"ID":"1ee479f4-3ca8-485a-ba93-b6cfa8808775","Type":"ContainerStarted","Data":"68add7603767fe2f1ffc0f648739b82ae69830ad224e3c9bd92a413d2a1959d6"} Sep 11 14:33:11.584350 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:11.584222 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk" Sep 11 14:33:11.586171 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:11.586146 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd" event={"ID":"f1c7493d-982c-44d9-b847-b930de23e261","Type":"ContainerStarted","Data":"db118c300959ba8584a449a81a0884b35c2d61bf447291a5be79e86f4dea41fd"} Sep 11 14:33:11.586330 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:11.586188 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd" event={"ID":"f1c7493d-982c-44d9-b847-b930de23e261","Type":"ContainerStarted","Data":"5934553c981eb0dbaaaf8360aa645659dc01549d6896722c4cc0b940c5e7772e"} Sep 11 14:33:11.586330 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:11.586210 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd" Sep 11 14:33:11.690559 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:11.690524 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:11.711902 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:11.711837 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd" podStartSLOduration=1.711817049 podStartE2EDuration="1.711817049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:33:11.710121034 +0000 UTC m=+1102.324227380" watchObservedRunningTime="2026-09-11 14:33:11.711817049 +0000 UTC m=+1102.325923395" Sep 11 14:33:11.750971 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:11.750911 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk" podStartSLOduration=1.75089567 podStartE2EDuration="1.75089567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:33:11.747505671 +0000 UTC m=+1102.361612017" watchObservedRunningTime="2026-09-11 14:33:11.75089567 +0000 UTC m=+1102.365002013" Sep 11 14:33:11.842209 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:11.842120 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:11.971028 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:11.970991 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:12.076890 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.076853 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:12.187205 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.187171 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:12.472676 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.472583 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:12.580083 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.580047 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:12.685365 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.685324 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w"] Sep 11 14:33:12.689031 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.689009 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w" Sep 11 14:33:12.695544 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.695527 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ac43663d-dockercfg-vrl4j\"" Sep 11 14:33:12.695724 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.695707 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ac43663d\"" Sep 11 14:33:12.707816 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.707794 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w"] Sep 11 14:33:12.783902 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.783814 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq"] Sep 11 14:33:12.784204 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.784184 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s8dm\" (UniqueName: \"kubernetes.io/projected/38480514-43fe-468c-8e46-8605ebc1af41-kube-api-access-5s8dm\") pod \"payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w\" (UID: \"38480514-43fe-468c-8e46-8605ebc1af41\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w" Sep 11 14:33:12.784251 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.784217 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38480514-43fe-468c-8e46-8605ebc1af41-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w\" (UID: \"38480514-43fe-468c-8e46-8605ebc1af41\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w" Sep 11 14:33:12.787358 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.787342 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq" Sep 11 14:33:12.813845 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.813806 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq"] Sep 11 14:33:12.885180 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.885145 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s8dm\" (UniqueName: \"kubernetes.io/projected/38480514-43fe-468c-8e46-8605ebc1af41-kube-api-access-5s8dm\") pod \"payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w\" (UID: \"38480514-43fe-468c-8e46-8605ebc1af41\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w" Sep 11 14:33:12.885180 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.885184 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c504ff2f-f9de-4309-9745-a360d20fdd88-plugins-config-volume\") pod \"payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq\" (UID: \"c504ff2f-f9de-4309-9745-a360d20fdd88\") " pod="openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq" Sep 11 14:33:12.885478 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.885210 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38480514-43fe-468c-8e46-8605ebc1af41-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w\" (UID: \"38480514-43fe-468c-8e46-8605ebc1af41\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w" Sep 11 14:33:12.885478 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.885240 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvr85\" (UniqueName: \"kubernetes.io/projected/c504ff2f-f9de-4309-9745-a360d20fdd88-kube-api-access-wvr85\") pod \"payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq\" (UID: \"c504ff2f-f9de-4309-9745-a360d20fdd88\") " pod="openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq" Sep 11 14:33:12.885980 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.885956 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38480514-43fe-468c-8e46-8605ebc1af41-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w\" (UID: \"38480514-43fe-468c-8e46-8605ebc1af41\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w" Sep 11 14:33:12.905372 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.905349 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s8dm\" (UniqueName: \"kubernetes.io/projected/38480514-43fe-468c-8e46-8605ebc1af41-kube-api-access-5s8dm\") pod \"payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w\" (UID: \"38480514-43fe-468c-8e46-8605ebc1af41\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w" Sep 11 14:33:12.986152 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.986118 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c504ff2f-f9de-4309-9745-a360d20fdd88-plugins-config-volume\") pod \"payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq\" (UID: \"c504ff2f-f9de-4309-9745-a360d20fdd88\") " pod="openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq" Sep 11 14:33:12.986152 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.986160 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvr85\" (UniqueName: \"kubernetes.io/projected/c504ff2f-f9de-4309-9745-a360d20fdd88-kube-api-access-wvr85\") pod \"payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq\" (UID: \"c504ff2f-f9de-4309-9745-a360d20fdd88\") " pod="openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq" Sep 11 14:33:12.986957 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.986935 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c504ff2f-f9de-4309-9745-a360d20fdd88-plugins-config-volume\") pod \"payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq\" (UID: \"c504ff2f-f9de-4309-9745-a360d20fdd88\") " pod="openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq" Sep 11 14:33:12.999301 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:12.999274 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w" Sep 11 14:33:13.001737 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:13.001714 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvr85\" (UniqueName: \"kubernetes.io/projected/c504ff2f-f9de-4309-9745-a360d20fdd88-kube-api-access-wvr85\") pod \"payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq\" (UID: \"c504ff2f-f9de-4309-9745-a360d20fdd88\") " pod="openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq" Sep 11 14:33:13.097149 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:13.097118 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq" Sep 11 14:33:13.223628 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:13.223585 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:13.301930 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:13.301845 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w"] Sep 11 14:33:13.304087 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:33:13.304060 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38480514_43fe_468c_8e46_8605ebc1af41.slice/crio-a06f9f809f2fffdae80b253b64de224e0057964ef5aa210e439899c2b185f69a WatchSource:0}: Error finding container a06f9f809f2fffdae80b253b64de224e0057964ef5aa210e439899c2b185f69a: Status 404 returned error can't find the container with id a06f9f809f2fffdae80b253b64de224e0057964ef5aa210e439899c2b185f69a Sep 11 14:33:13.427917 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:13.427895 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq"] Sep 11 14:33:13.430272 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:33:13.430240 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc504ff2f_f9de_4309_9745_a360d20fdd88.slice/crio-90b545f8496d25cbcc457738aab07a947434d08875e192d37ff5cfb5253891c9 WatchSource:0}: Error finding container 90b545f8496d25cbcc457738aab07a947434d08875e192d37ff5cfb5253891c9: Status 404 returned error can't find the container with id 90b545f8496d25cbcc457738aab07a947434d08875e192d37ff5cfb5253891c9 Sep 11 14:33:13.598305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:13.598209 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq" event={"ID":"c504ff2f-f9de-4309-9745-a360d20fdd88","Type":"ContainerStarted","Data":"d2466cca8234f276565f27bbc325534f6c870d02dd98f02f6b31ca58330c470d"} Sep 11 14:33:13.598305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:13.598270 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq" event={"ID":"c504ff2f-f9de-4309-9745-a360d20fdd88","Type":"ContainerStarted","Data":"90b545f8496d25cbcc457738aab07a947434d08875e192d37ff5cfb5253891c9"} Sep 11 14:33:13.598305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:13.598302 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq" Sep 11 14:33:13.599965 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:13.599941 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w" event={"ID":"38480514-43fe-468c-8e46-8605ebc1af41","Type":"ContainerStarted","Data":"13bfd81612903790f1fb95f9f000b7884adf7fa3453b357fd70ef6eb06fbff3c"} Sep 11 14:33:13.600109 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:13.599985 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w" event={"ID":"38480514-43fe-468c-8e46-8605ebc1af41","Type":"ContainerStarted","Data":"a06f9f809f2fffdae80b253b64de224e0057964ef5aa210e439899c2b185f69a"} Sep 11 14:33:13.600109 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:13.600014 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w" Sep 11 14:33:13.647532 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:13.647459 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq" podStartSLOduration=1.647406717 podStartE2EDuration="1.647406717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:33:13.64471552 +0000 UTC m=+1104.258821889" watchObservedRunningTime="2026-09-11 14:33:13.647406717 +0000 UTC m=+1104.261513062" Sep 11 14:33:13.981690 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:13.981645 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:14.120074 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:14.120034 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:14.274116 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:14.274030 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:15.094288 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:15.094251 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:15.517405 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:15.517374 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:16.710632 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:16.710598 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:17.783626 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:17.783597 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:18.302591 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:18.302524 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:19.903342 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:19.903309 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:20.034840 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:20.034809 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:20.352809 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:20.352707 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:21.606226 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:21.606183 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:21.804972 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:21.804927 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:22.593096 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:22.593063 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk" Sep 11 14:33:22.593306 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:22.593133 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd" Sep 11 14:33:22.728498 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:22.728443 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w" podStartSLOduration=10.728426988 podStartE2EDuration="10.728426988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:33:13.701719381 +0000 UTC m=+1104.315825728" watchObservedRunningTime="2026-09-11 14:33:22.728426988 +0000 UTC m=+1113.342533322" Sep 11 14:33:22.901273 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:22.901245 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8"] Sep 11 14:33:22.904972 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:22.904951 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8" Sep 11 14:33:22.911572 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:33:22.911539 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="d25ca92e-bd88-4c25-a601-16fc41c3fcd3" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8" Sep 11 14:33:22.912236 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:33:22.912022 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"payload-processing-plugins-e2e-inf-b-5b0917\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5b0917\"" type="*v1.ConfigMap" Sep 11 14:33:22.918744 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:33:22.918709 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"payload-processing-e2e-inf-b-5b0917-dockercfg-wb94z\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5b0917-dockercfg-wb94z\"" type="*v1.Secret" Sep 11 14:33:22.965602 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:22.965573 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8"] Sep 11 14:33:23.083631 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:23.083587 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25ca92e-bd88-4c25-a601-16fc41c3fcd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8\" (UID: \"d25ca92e-bd88-4c25-a601-16fc41c3fcd3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8" Sep 11 14:33:23.083827 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:23.083689 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwf4w\" (UniqueName: \"kubernetes.io/projected/d25ca92e-bd88-4c25-a601-16fc41c3fcd3-kube-api-access-hwf4w\") pod \"payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8\" (UID: \"d25ca92e-bd88-4c25-a601-16fc41c3fcd3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8" Sep 11 14:33:23.184622 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:23.184535 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25ca92e-bd88-4c25-a601-16fc41c3fcd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8\" (UID: \"d25ca92e-bd88-4c25-a601-16fc41c3fcd3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8" Sep 11 14:33:23.184789 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:23.184622 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwf4w\" (UniqueName: \"kubernetes.io/projected/d25ca92e-bd88-4c25-a601-16fc41c3fcd3-kube-api-access-hwf4w\") pod \"payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8\" (UID: \"d25ca92e-bd88-4c25-a601-16fc41c3fcd3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8" Sep 11 14:33:23.204710 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:23.204682 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwf4w\" (UniqueName: \"kubernetes.io/projected/d25ca92e-bd88-4c25-a601-16fc41c3fcd3-kube-api-access-hwf4w\") pod \"payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8\" (UID: \"d25ca92e-bd88-4c25-a601-16fc41c3fcd3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8" Sep 11 14:33:23.597004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:23.596917 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:23.794703 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:23.794668 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:24.057906 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:24.057867 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5b0917-dockercfg-wb94z\"" Sep 11 14:33:24.185035 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:33:24.184997 2553 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-5b0917: failed to sync configmap cache: timed out waiting for the condition Sep 11 14:33:24.185199 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:33:24.185077 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d25ca92e-bd88-4c25-a601-16fc41c3fcd3-plugins-config-volume podName:d25ca92e-bd88-4c25-a601-16fc41c3fcd3 nodeName:}" failed. No retries permitted until 2026-09-11 14:33:24.685060498 +0000 UTC m=+1115.299166820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/d25ca92e-bd88-4c25-a601-16fc41c3fcd3-plugins-config-volume") pod "payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8" (UID: "d25ca92e-bd88-4c25-a601-16fc41c3fcd3") : failed to sync configmap cache: timed out waiting for the condition Sep 11 14:33:24.480063 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:24.480029 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5b0917\"" Sep 11 14:33:24.607592 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:24.607562 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq" Sep 11 14:33:24.607776 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:24.607629 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w" Sep 11 14:33:24.697977 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:24.697940 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25ca92e-bd88-4c25-a601-16fc41c3fcd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8\" (UID: \"d25ca92e-bd88-4c25-a601-16fc41c3fcd3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8" Sep 11 14:33:24.698698 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:24.698680 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25ca92e-bd88-4c25-a601-16fc41c3fcd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8\" (UID: \"d25ca92e-bd88-4c25-a601-16fc41c3fcd3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8" Sep 11 14:33:24.715465 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:24.715434 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8" Sep 11 14:33:25.031761 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:25.031739 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8"] Sep 11 14:33:25.034376 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:33:25.034337 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25ca92e_bd88_4c25_a601_16fc41c3fcd3.slice/crio-7cab169311032a7311ad72cd2a5c98e4e734e18102d59d7ddfb044934cfae9d4 WatchSource:0}: Error finding container 7cab169311032a7311ad72cd2a5c98e4e734e18102d59d7ddfb044934cfae9d4: Status 404 returned error can't find the container with id 7cab169311032a7311ad72cd2a5c98e4e734e18102d59d7ddfb044934cfae9d4 Sep 11 14:33:25.346446 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:25.346343 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:25.661377 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:25.661340 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8" event={"ID":"d25ca92e-bd88-4c25-a601-16fc41c3fcd3","Type":"ContainerStarted","Data":"0e972de7a56baaf9d0ae7349d193a8fbb33d3cd1701812bead46f26cea8c265c"} Sep 11 14:33:25.661595 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:25.661574 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8" Sep 11 14:33:25.661646 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:25.661628 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8" event={"ID":"d25ca92e-bd88-4c25-a601-16fc41c3fcd3","Type":"ContainerStarted","Data":"7cab169311032a7311ad72cd2a5c98e4e734e18102d59d7ddfb044934cfae9d4"} Sep 11 14:33:25.725075 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:25.725023 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8" podStartSLOduration=3.725008012 podStartE2EDuration="3.725008012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:33:25.713031331 +0000 UTC m=+1116.327137673" watchObservedRunningTime="2026-09-11 14:33:25.725008012 +0000 UTC m=+1116.339114387" Sep 11 14:33:27.196204 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:27.196159 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:29.449023 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:29.448993 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:31.169064 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:31.169017 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:36.669371 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:36.669337 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8" Sep 11 14:33:47.941879 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:47.941848 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:48.293534 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:48.293450 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:49.210637 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:49.210600 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:49.451858 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:49.451819 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:49.652626 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:49.652589 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:49.983312 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:49.983255 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:51.593054 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:51.593017 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:53.386364 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:53.386325 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:54.455871 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:54.455832 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:55.019457 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:55.019402 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:56.196649 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:56.196610 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:56.593259 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:56.593172 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:57.990006 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:57.989961 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:58.574266 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:58.574222 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:58.695337 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:58.695288 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:58.842510 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:58.842408 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:59.033983 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.033943 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:59.160621 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.160588 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:59.292553 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.292524 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq"] Sep 11 14:33:59.292788 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.292766 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq" podUID="c504ff2f-f9de-4309-9745-a360d20fdd88" containerName="payload-processing" containerID="cri-o://d2466cca8234f276565f27bbc325534f6c870d02dd98f02f6b31ca58330c470d" gracePeriod=30 Sep 11 14:33:59.318073 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.318047 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w"] Sep 11 14:33:59.318266 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.318246 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w" podUID="38480514-43fe-468c-8e46-8605ebc1af41" containerName="payload-pre-processing" containerID="cri-o://13bfd81612903790f1fb95f9f000b7884adf7fa3453b357fd70ef6eb06fbff3c" gracePeriod=30 Sep 11 14:33:59.374997 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.374970 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:59.528793 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.528688 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:59.565915 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.565892 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq" Sep 11 14:33:59.595128 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.595104 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w" Sep 11 14:33:59.728578 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.728503 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38480514-43fe-468c-8e46-8605ebc1af41-plugins-config-volume\") pod \"38480514-43fe-468c-8e46-8605ebc1af41\" (UID: \"38480514-43fe-468c-8e46-8605ebc1af41\") " Sep 11 14:33:59.728578 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.728538 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5s8dm\" (UniqueName: \"kubernetes.io/projected/38480514-43fe-468c-8e46-8605ebc1af41-kube-api-access-5s8dm\") pod \"38480514-43fe-468c-8e46-8605ebc1af41\" (UID: \"38480514-43fe-468c-8e46-8605ebc1af41\") " Sep 11 14:33:59.728761 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.728599 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvr85\" (UniqueName: \"kubernetes.io/projected/c504ff2f-f9de-4309-9745-a360d20fdd88-kube-api-access-wvr85\") pod \"c504ff2f-f9de-4309-9745-a360d20fdd88\" (UID: \"c504ff2f-f9de-4309-9745-a360d20fdd88\") " Sep 11 14:33:59.728761 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.728625 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c504ff2f-f9de-4309-9745-a360d20fdd88-plugins-config-volume\") pod \"c504ff2f-f9de-4309-9745-a360d20fdd88\" (UID: \"c504ff2f-f9de-4309-9745-a360d20fdd88\") " Sep 11 14:33:59.728935 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.728912 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38480514-43fe-468c-8e46-8605ebc1af41-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "38480514-43fe-468c-8e46-8605ebc1af41" (UID: "38480514-43fe-468c-8e46-8605ebc1af41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:33:59.729045 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.729026 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c504ff2f-f9de-4309-9745-a360d20fdd88-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c504ff2f-f9de-4309-9745-a360d20fdd88" (UID: "c504ff2f-f9de-4309-9745-a360d20fdd88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:33:59.730869 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.730845 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38480514-43fe-468c-8e46-8605ebc1af41-kube-api-access-5s8dm" (OuterVolumeSpecName: "kube-api-access-5s8dm") pod "38480514-43fe-468c-8e46-8605ebc1af41" (UID: "38480514-43fe-468c-8e46-8605ebc1af41"). InnerVolumeSpecName "kube-api-access-5s8dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:33:59.730932 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.730851 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c504ff2f-f9de-4309-9745-a360d20fdd88-kube-api-access-wvr85" (OuterVolumeSpecName: "kube-api-access-wvr85") pod "c504ff2f-f9de-4309-9745-a360d20fdd88" (UID: "c504ff2f-f9de-4309-9745-a360d20fdd88"). InnerVolumeSpecName "kube-api-access-wvr85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:33:59.741031 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.740992 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:59.829544 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.829514 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38480514-43fe-468c-8e46-8605ebc1af41-plugins-config-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:33:59.829544 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.829539 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s8dm\" (UniqueName: \"kubernetes.io/projected/38480514-43fe-468c-8e46-8605ebc1af41-kube-api-access-5s8dm\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:33:59.829544 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.829553 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvr85\" (UniqueName: \"kubernetes.io/projected/c504ff2f-f9de-4309-9745-a360d20fdd88-kube-api-access-wvr85\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:33:59.829850 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.829562 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c504ff2f-f9de-4309-9745-a360d20fdd88-plugins-config-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:33:59.832037 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.832001 2553 generic.go:356] "Generic (PLEG): container finished" podID="c504ff2f-f9de-4309-9745-a360d20fdd88" containerID="d2466cca8234f276565f27bbc325534f6c870d02dd98f02f6b31ca58330c470d" exitCode=0 Sep 11 14:33:59.832198 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.832062 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq" Sep 11 14:33:59.832198 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.832085 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq" event={"ID":"c504ff2f-f9de-4309-9745-a360d20fdd88","Type":"ContainerDied","Data":"d2466cca8234f276565f27bbc325534f6c870d02dd98f02f6b31ca58330c470d"} Sep 11 14:33:59.832198 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.832134 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq" event={"ID":"c504ff2f-f9de-4309-9745-a360d20fdd88","Type":"ContainerDied","Data":"90b545f8496d25cbcc457738aab07a947434d08875e192d37ff5cfb5253891c9"} Sep 11 14:33:59.832198 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.832161 2553 scope.go:117] "RemoveContainer" containerID="d2466cca8234f276565f27bbc325534f6c870d02dd98f02f6b31ca58330c470d" Sep 11 14:33:59.833403 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.833379 2553 generic.go:356] "Generic (PLEG): container finished" podID="38480514-43fe-468c-8e46-8605ebc1af41" containerID="13bfd81612903790f1fb95f9f000b7884adf7fa3453b357fd70ef6eb06fbff3c" exitCode=0 Sep 11 14:33:59.833513 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.833424 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w" Sep 11 14:33:59.833513 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.833475 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w" event={"ID":"38480514-43fe-468c-8e46-8605ebc1af41","Type":"ContainerDied","Data":"13bfd81612903790f1fb95f9f000b7884adf7fa3453b357fd70ef6eb06fbff3c"} Sep 11 14:33:59.833628 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.833529 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w" event={"ID":"38480514-43fe-468c-8e46-8605ebc1af41","Type":"ContainerDied","Data":"a06f9f809f2fffdae80b253b64de224e0057964ef5aa210e439899c2b185f69a"} Sep 11 14:33:59.841544 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.841130 2553 scope.go:117] "RemoveContainer" containerID="d2466cca8234f276565f27bbc325534f6c870d02dd98f02f6b31ca58330c470d" Sep 11 14:33:59.841544 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:33:59.841482 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2466cca8234f276565f27bbc325534f6c870d02dd98f02f6b31ca58330c470d\": container with ID starting with d2466cca8234f276565f27bbc325534f6c870d02dd98f02f6b31ca58330c470d not found: ID does not exist" containerID="d2466cca8234f276565f27bbc325534f6c870d02dd98f02f6b31ca58330c470d" Sep 11 14:33:59.841544 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.841514 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2466cca8234f276565f27bbc325534f6c870d02dd98f02f6b31ca58330c470d"} err="failed to get container status \"d2466cca8234f276565f27bbc325534f6c870d02dd98f02f6b31ca58330c470d\": rpc error: code = NotFound desc = could not find container \"d2466cca8234f276565f27bbc325534f6c870d02dd98f02f6b31ca58330c470d\": container with ID starting with d2466cca8234f276565f27bbc325534f6c870d02dd98f02f6b31ca58330c470d not found: ID does not exist" Sep 11 14:33:59.841544 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.841532 2553 scope.go:117] "RemoveContainer" containerID="13bfd81612903790f1fb95f9f000b7884adf7fa3453b357fd70ef6eb06fbff3c" Sep 11 14:33:59.845985 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.845960 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:33:59.852034 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.852013 2553 scope.go:117] "RemoveContainer" containerID="13bfd81612903790f1fb95f9f000b7884adf7fa3453b357fd70ef6eb06fbff3c" Sep 11 14:33:59.852313 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:33:59.852295 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13bfd81612903790f1fb95f9f000b7884adf7fa3453b357fd70ef6eb06fbff3c\": container with ID starting with 13bfd81612903790f1fb95f9f000b7884adf7fa3453b357fd70ef6eb06fbff3c not found: ID does not exist" containerID="13bfd81612903790f1fb95f9f000b7884adf7fa3453b357fd70ef6eb06fbff3c" Sep 11 14:33:59.852374 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.852321 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13bfd81612903790f1fb95f9f000b7884adf7fa3453b357fd70ef6eb06fbff3c"} err="failed to get container status \"13bfd81612903790f1fb95f9f000b7884adf7fa3453b357fd70ef6eb06fbff3c\": rpc error: code = NotFound desc = could not find container \"13bfd81612903790f1fb95f9f000b7884adf7fa3453b357fd70ef6eb06fbff3c\": container with ID starting with 13bfd81612903790f1fb95f9f000b7884adf7fa3453b357fd70ef6eb06fbff3c not found: ID does not exist" Sep 11 14:33:59.880558 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.880530 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w"] Sep 11 14:33:59.895584 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.895554 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ac43663d-86947d7fd6-nsp5w"] Sep 11 14:33:59.913109 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.913081 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38480514-43fe-468c-8e46-8605ebc1af41" path="/var/lib/kubelet/pods/38480514-43fe-468c-8e46-8605ebc1af41/volumes" Sep 11 14:33:59.931959 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.931928 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq"] Sep 11 14:33:59.933676 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:33:59.933653 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ac43663d-7f7df95fff-mm6mq"] Sep 11 14:34:00.067360 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:00.067253 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:00.512480 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:00.512404 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:00.675952 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:00.675917 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:00.771518 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:00.771434 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:00.874799 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:00.874762 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:01.247842 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.247806 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss"] Sep 11 14:34:01.248458 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.248436 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38480514-43fe-468c-8e46-8605ebc1af41" containerName="payload-pre-processing" Sep 11 14:34:01.248532 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.248460 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="38480514-43fe-468c-8e46-8605ebc1af41" containerName="payload-pre-processing" Sep 11 14:34:01.248532 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.248472 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c504ff2f-f9de-4309-9745-a360d20fdd88" containerName="payload-processing" Sep 11 14:34:01.248532 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.248482 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c504ff2f-f9de-4309-9745-a360d20fdd88" containerName="payload-processing" Sep 11 14:34:01.248692 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.248596 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="38480514-43fe-468c-8e46-8605ebc1af41" containerName="payload-pre-processing" Sep 11 14:34:01.248692 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.248609 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c504ff2f-f9de-4309-9745-a360d20fdd88" containerName="payload-processing" Sep 11 14:34:01.253470 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.253451 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.257975 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.257954 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 14:34:01.258142 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.258120 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-d7531909-gw-openshift-default-dockercfg-ktz8m\"" Sep 11 14:34:01.300685 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.300652 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss"] Sep 11 14:34:01.343852 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.343811 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmpgj\" (UniqueName: \"kubernetes.io/projected/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-kube-api-access-rmpgj\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.344057 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.343867 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-podinfo\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.344057 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.343898 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-envoy\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.344057 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.343931 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-data\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.344057 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.343951 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-token\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.344057 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.343974 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-workload-certs\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.344057 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.344037 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-credential-socket\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.344382 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.344061 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istiod-ca-cert\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.344382 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.344088 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-workload-socket\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.391936 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.391901 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:01.445527 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.445483 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmpgj\" (UniqueName: \"kubernetes.io/projected/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-kube-api-access-rmpgj\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.445732 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.445552 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-podinfo\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.445732 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.445584 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-envoy\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.445732 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.445620 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-data\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.445732 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.445658 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-token\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.445732 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.445689 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-workload-certs\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.446016 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.445756 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-credential-socket\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.446076 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.446010 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istiod-ca-cert\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.446076 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.446051 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-workload-socket\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.446170 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.446103 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-data\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.446170 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.446144 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-workload-certs\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.446280 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.446178 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-credential-socket\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.446371 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.446350 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-workload-socket\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.446791 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.446771 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istiod-ca-cert\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.448378 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.448359 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-envoy\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.448604 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.448584 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-podinfo\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.490961 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.490931 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-token\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.491203 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.491179 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmpgj\" (UniqueName: \"kubernetes.io/projected/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-kube-api-access-rmpgj\") pod \"e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.564268 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.564163 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:01.913588 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:34:01.913537 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf39c37f6_0d79_49fc_a78a_7f06948d3ef0.slice/crio-a328e0ec5363611c6c47f2673179686054fead975fbe85e1055ed80fe1271f8a WatchSource:0}: Error finding container a328e0ec5363611c6c47f2673179686054fead975fbe85e1055ed80fe1271f8a: Status 404 returned error can't find the container with id a328e0ec5363611c6c47f2673179686054fead975fbe85e1055ed80fe1271f8a Sep 11 14:34:01.916191 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.916156 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:34:01.920600 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.920574 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c504ff2f-f9de-4309-9745-a360d20fdd88" path="/var/lib/kubelet/pods/c504ff2f-f9de-4309-9745-a360d20fdd88/volumes" Sep 11 14:34:01.920843 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:01.920829 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss"] Sep 11 14:34:02.102586 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:02.102546 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:02.814756 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:02.814713 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:02.855493 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:02.855456 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" event={"ID":"f39c37f6-0d79-49fc-a78a-7f06948d3ef0","Type":"ContainerStarted","Data":"a328e0ec5363611c6c47f2673179686054fead975fbe85e1055ed80fe1271f8a"} Sep 11 14:34:04.460866 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:04.460828 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:34:04.461177 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:04.460898 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:34:04.461177 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:04.460930 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:34:04.869500 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:04.869393 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" event={"ID":"f39c37f6-0d79-49fc-a78a-7f06948d3ef0","Type":"ContainerStarted","Data":"1693124b05f3dce6908536c81768d951bd53f2256be5b06a9db0af71a35dd35e"} Sep 11 14:34:04.946488 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:04.946437 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" podStartSLOduration=1.402262981 podStartE2EDuration="3.946402736s" totalImagesPullingTime="2.544139755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:34:04.945867273 +0000 UTC m=+1155.559973618" watchObservedRunningTime="2026-09-11 14:34:04.946402736 +0000 UTC m=+1155.560509079" Sep 11 14:34:05.029385 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:05.029352 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:05.564756 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:05.564725 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:05.571567 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:05.571539 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:05.874723 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:05.874645 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:05.875536 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:05.875518 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:06.408794 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.408757 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6"] Sep 11 14:34:06.412645 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.412628 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:06.426378 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.424430 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c322a2\"/\"kube-root-ca.crt\"" Sep 11 14:34:06.426378 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.424604 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c322a2\"/\"autores-model-c322a2-kserve-self-signed-certs\"" Sep 11 14:34:06.426378 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.424610 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c322a2\"/\"default-dockercfg-hvf6d\"" Sep 11 14:34:06.426378 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.424668 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c322a2\"/\"openshift-service-ca.crt\"" Sep 11 14:34:06.426919 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.426897 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:06.492486 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.492449 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6"] Sep 11 14:34:06.597914 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.597880 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-home\") pod \"autores-model-c322a2-kserve-686b9976c5-45tk6\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:06.598340 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.597926 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-model-cache\") pod \"autores-model-c322a2-kserve-686b9976c5-45tk6\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:06.598340 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.597949 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a01974-56cc-45fa-b284-922c51bbf554-tls-certs\") pod \"autores-model-c322a2-kserve-686b9976c5-45tk6\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:06.598340 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.598118 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgq42\" (UniqueName: \"kubernetes.io/projected/f6a01974-56cc-45fa-b284-922c51bbf554-kube-api-access-fgq42\") pod \"autores-model-c322a2-kserve-686b9976c5-45tk6\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:06.598340 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.598241 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-dshm\") pod \"autores-model-c322a2-kserve-686b9976c5-45tk6\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:06.598340 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.598276 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-tmp-dir\") pod \"autores-model-c322a2-kserve-686b9976c5-45tk6\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:06.699468 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.699365 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgq42\" (UniqueName: \"kubernetes.io/projected/f6a01974-56cc-45fa-b284-922c51bbf554-kube-api-access-fgq42\") pod \"autores-model-c322a2-kserve-686b9976c5-45tk6\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:06.699644 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.699522 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-dshm\") pod \"autores-model-c322a2-kserve-686b9976c5-45tk6\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:06.699644 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.699548 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-tmp-dir\") pod \"autores-model-c322a2-kserve-686b9976c5-45tk6\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:06.699644 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.699612 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-home\") pod \"autores-model-c322a2-kserve-686b9976c5-45tk6\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:06.699811 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.699647 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-model-cache\") pod \"autores-model-c322a2-kserve-686b9976c5-45tk6\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:06.699811 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.699673 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a01974-56cc-45fa-b284-922c51bbf554-tls-certs\") pod \"autores-model-c322a2-kserve-686b9976c5-45tk6\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:06.700128 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.700105 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-home\") pod \"autores-model-c322a2-kserve-686b9976c5-45tk6\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:06.700324 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.700296 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-tmp-dir\") pod \"autores-model-c322a2-kserve-686b9976c5-45tk6\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:06.700555 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.700527 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-model-cache\") pod \"autores-model-c322a2-kserve-686b9976c5-45tk6\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:06.702074 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.702049 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-dshm\") pod \"autores-model-c322a2-kserve-686b9976c5-45tk6\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:06.702331 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.702309 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a01974-56cc-45fa-b284-922c51bbf554-tls-certs\") pod \"autores-model-c322a2-kserve-686b9976c5-45tk6\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:06.714493 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.714472 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgq42\" (UniqueName: \"kubernetes.io/projected/f6a01974-56cc-45fa-b284-922c51bbf554-kube-api-access-fgq42\") pod \"autores-model-c322a2-kserve-686b9976c5-45tk6\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:06.722405 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:06.722382 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:07.021576 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:07.021551 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6"] Sep 11 14:34:07.023559 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:34:07.023535 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6a01974_56cc_45fa_b284_922c51bbf554.slice/crio-a4a6ce85c5934a6ddb222f995e841e726ac5871a7bd3773603e5db6e369acce9 WatchSource:0}: Error finding container a4a6ce85c5934a6ddb222f995e841e726ac5871a7bd3773603e5db6e369acce9: Status 404 returned error can't find the container with id a4a6ce85c5934a6ddb222f995e841e726ac5871a7bd3773603e5db6e369acce9 Sep 11 14:34:07.195206 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:07.195178 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:07.405727 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:07.405676 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:07.887085 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:07.887002 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" event={"ID":"f6a01974-56cc-45fa-b284-922c51bbf554","Type":"ContainerStarted","Data":"551d8e1f96714abe123e5c3d4deb97dac5d5af70c82b88d5abc596e63da5a25d"} Sep 11 14:34:07.887085 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:07.887046 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" event={"ID":"f6a01974-56cc-45fa-b284-922c51bbf554","Type":"ContainerStarted","Data":"a4a6ce85c5934a6ddb222f995e841e726ac5871a7bd3773603e5db6e369acce9"} Sep 11 14:34:07.969464 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:07.969390 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" podStartSLOduration=1.9693719079999998 podStartE2EDuration="1.969371908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:34:07.968133133 +0000 UTC m=+1158.582239477" watchObservedRunningTime="2026-09-11 14:34:07.969371908 +0000 UTC m=+1158.583478251" Sep 11 14:34:08.297679 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:08.297642 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:09.836993 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:09.836959 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:09.951868 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:09.951832 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:10.323381 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:10.323298 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:10.597456 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:10.597349 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:10.723437 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:10.723381 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:11.197711 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:11.197681 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:11.628294 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:11.628175 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:12.074391 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:12.074262 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx"] Sep 11 14:34:12.078825 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:12.078798 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx" Sep 11 14:34:12.082096 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:12.082077 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d7531909\"" Sep 11 14:34:12.082719 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:12.082701 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d7531909-dockercfg-rbn7n\"" Sep 11 14:34:12.111349 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:12.111313 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx"] Sep 11 14:34:12.143698 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:12.143647 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpl6b\" (UniqueName: \"kubernetes.io/projected/14f7e9e8-4592-440a-be97-18931f29f144-kube-api-access-hpl6b\") pod \"payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx\" (UID: \"14f7e9e8-4592-440a-be97-18931f29f144\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx" Sep 11 14:34:12.143907 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:12.143854 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14f7e9e8-4592-440a-be97-18931f29f144-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx\" (UID: \"14f7e9e8-4592-440a-be97-18931f29f144\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx" Sep 11 14:34:12.244716 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:12.244681 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpl6b\" (UniqueName: \"kubernetes.io/projected/14f7e9e8-4592-440a-be97-18931f29f144-kube-api-access-hpl6b\") pod \"payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx\" (UID: \"14f7e9e8-4592-440a-be97-18931f29f144\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx" Sep 11 14:34:12.245176 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:12.244836 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14f7e9e8-4592-440a-be97-18931f29f144-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx\" (UID: \"14f7e9e8-4592-440a-be97-18931f29f144\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx" Sep 11 14:34:12.245618 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:12.245593 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14f7e9e8-4592-440a-be97-18931f29f144-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx\" (UID: \"14f7e9e8-4592-440a-be97-18931f29f144\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx" Sep 11 14:34:12.260895 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:12.260864 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpl6b\" (UniqueName: \"kubernetes.io/projected/14f7e9e8-4592-440a-be97-18931f29f144-kube-api-access-hpl6b\") pod \"payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx\" (UID: \"14f7e9e8-4592-440a-be97-18931f29f144\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx" Sep 11 14:34:12.390094 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:12.390065 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx" Sep 11 14:34:12.679333 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:12.679253 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx"] Sep 11 14:34:12.682650 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:34:12.682626 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f7e9e8_4592_440a_be97_18931f29f144.slice/crio-7a51b07c5581a01862d7637ca943451df22b62d2852ff6cb3fd74d157ffcda3d WatchSource:0}: Error finding container 7a51b07c5581a01862d7637ca943451df22b62d2852ff6cb3fd74d157ffcda3d: Status 404 returned error can't find the container with id 7a51b07c5581a01862d7637ca943451df22b62d2852ff6cb3fd74d157ffcda3d Sep 11 14:34:12.694947 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:12.694922 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:12.916953 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:12.916912 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx" event={"ID":"14f7e9e8-4592-440a-be97-18931f29f144","Type":"ContainerStarted","Data":"55783a32806c28c56fe94fbefa68fee24dd8a084a0ed2cbb44661b8ec96b645b"} Sep 11 14:34:12.917128 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:12.916981 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx" Sep 11 14:34:12.917128 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:12.917009 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx" event={"ID":"14f7e9e8-4592-440a-be97-18931f29f144","Type":"ContainerStarted","Data":"7a51b07c5581a01862d7637ca943451df22b62d2852ff6cb3fd74d157ffcda3d"} Sep 11 14:34:13.997685 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:13.997652 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:15.201716 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:15.201679 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:16.595875 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:16.595835 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:16.723084 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:16.723044 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:16.723270 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:16.723108 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:16.736044 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:16.736011 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:16.779075 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:16.779027 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx" podStartSLOduration=4.779010692 podStartE2EDuration="4.779010692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:34:12.952156923 +0000 UTC m=+1163.566263268" watchObservedRunningTime="2026-09-11 14:34:16.779010692 +0000 UTC m=+1167.393117036" Sep 11 14:34:16.956262 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:16.956235 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:19.627409 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:19.627373 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:20.579491 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:20.579449 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:22.795562 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:22.795529 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:23.924183 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:23.924158 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx" Sep 11 14:34:24.142449 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:24.142397 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:25.459710 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:25.459661 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:26.881708 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:26.881666 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:27.906541 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:27.906504 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:28.008600 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:28.008564 2553 generic.go:356] "Generic (PLEG): container finished" podID="f39c37f6-0d79-49fc-a78a-7f06948d3ef0" containerID="1693124b05f3dce6908536c81768d951bd53f2256be5b06a9db0af71a35dd35e" exitCode=0 Sep 11 14:34:28.008600 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:28.008605 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" event={"ID":"f39c37f6-0d79-49fc-a78a-7f06948d3ef0","Type":"ContainerDied","Data":"1693124b05f3dce6908536c81768d951bd53f2256be5b06a9db0af71a35dd35e"} Sep 11 14:34:28.009002 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:28.008987 2553 scope.go:117] "RemoveContainer" containerID="1693124b05f3dce6908536c81768d951bd53f2256be5b06a9db0af71a35dd35e" Sep 11 14:34:28.010161 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:28.010129 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:34:28.010250 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:28.010195 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:34:28.010250 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:28.010229 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:34:29.015012 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:29.014982 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" event={"ID":"f39c37f6-0d79-49fc-a78a-7f06948d3ef0","Type":"ContainerStarted","Data":"c64bc3915b735c132ce9ffb5457ce9e48f10186ef8129efd1799f84a1946edcd"} Sep 11 14:34:29.565111 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:29.565074 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:29.565707 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:29.565607 2553 patch_prober.go:28] interesting pod/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 11 14:34:29.565906 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:29.565881 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" podUID="f39c37f6-0d79-49fc-a78a-7f06948d3ef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 11 14:34:30.564877 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:30.564836 2553 patch_prober.go:28] interesting pod/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 11 14:34:30.565533 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:30.565498 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" podUID="f39c37f6-0d79-49fc-a78a-7f06948d3ef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 11 14:34:31.574933 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:31.574884 2553 patch_prober.go:28] interesting pod/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 11 14:34:31.575319 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:31.574967 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" podUID="f39c37f6-0d79-49fc-a78a-7f06948d3ef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 11 14:34:31.584664 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:31.584625 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:32.213040 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:32.212999 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:32.564737 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:32.564634 2553 patch_prober.go:28] interesting pod/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 11 14:34:32.564737 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:32.564706 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" podUID="f39c37f6-0d79-49fc-a78a-7f06948d3ef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 11 14:34:33.259033 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:33.258977 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:33.322486 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:33.322456 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx"] Sep 11 14:34:33.322704 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:33.322682 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx" podUID="14f7e9e8-4592-440a-be97-18931f29f144" containerName="payload-pre-processing" containerID="cri-o://55783a32806c28c56fe94fbefa68fee24dd8a084a0ed2cbb44661b8ec96b645b" gracePeriod=30 Sep 11 14:34:33.367437 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:33.363504 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:33.442466 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:33.442432 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:33.568273 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:33.567814 2553 patch_prober.go:28] interesting pod/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Sep 11 14:34:33.568273 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:33.567877 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" podUID="f39c37f6-0d79-49fc-a78a-7f06948d3ef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Sep 11 14:34:33.576869 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:33.576835 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:33.632173 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:33.632150 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx" Sep 11 14:34:33.718944 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:33.718904 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:33.781207 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:33.781112 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpl6b\" (UniqueName: \"kubernetes.io/projected/14f7e9e8-4592-440a-be97-18931f29f144-kube-api-access-hpl6b\") pod \"14f7e9e8-4592-440a-be97-18931f29f144\" (UID: \"14f7e9e8-4592-440a-be97-18931f29f144\") " Sep 11 14:34:33.781383 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:33.781248 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14f7e9e8-4592-440a-be97-18931f29f144-plugins-config-volume\") pod \"14f7e9e8-4592-440a-be97-18931f29f144\" (UID: \"14f7e9e8-4592-440a-be97-18931f29f144\") " Sep 11 14:34:33.781922 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:33.781887 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14f7e9e8-4592-440a-be97-18931f29f144-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "14f7e9e8-4592-440a-be97-18931f29f144" (UID: "14f7e9e8-4592-440a-be97-18931f29f144"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:34:33.784027 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:33.783992 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f7e9e8-4592-440a-be97-18931f29f144-kube-api-access-hpl6b" (OuterVolumeSpecName: "kube-api-access-hpl6b") pod "14f7e9e8-4592-440a-be97-18931f29f144" (UID: "14f7e9e8-4592-440a-be97-18931f29f144"). InnerVolumeSpecName "kube-api-access-hpl6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:34:33.797161 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:33.797132 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss"] Sep 11 14:34:33.797363 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:33.797336 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" podUID="f39c37f6-0d79-49fc-a78a-7f06948d3ef0" containerName="istio-proxy" containerID="cri-o://c64bc3915b735c132ce9ffb5457ce9e48f10186ef8129efd1799f84a1946edcd" gracePeriod=30 Sep 11 14:34:33.866441 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:33.863279 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:33.882739 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:33.882702 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14f7e9e8-4592-440a-be97-18931f29f144-plugins-config-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:34:33.882739 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:33.882729 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpl6b\" (UniqueName: \"kubernetes.io/projected/14f7e9e8-4592-440a-be97-18931f29f144-kube-api-access-hpl6b\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:34:34.041893 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:34.041798 2553 generic.go:356] "Generic (PLEG): container finished" podID="14f7e9e8-4592-440a-be97-18931f29f144" containerID="55783a32806c28c56fe94fbefa68fee24dd8a084a0ed2cbb44661b8ec96b645b" exitCode=0 Sep 11 14:34:34.041893 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:34.041849 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx" Sep 11 14:34:34.041893 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:34.041888 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx" event={"ID":"14f7e9e8-4592-440a-be97-18931f29f144","Type":"ContainerDied","Data":"55783a32806c28c56fe94fbefa68fee24dd8a084a0ed2cbb44661b8ec96b645b"} Sep 11 14:34:34.042173 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:34.041948 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx" event={"ID":"14f7e9e8-4592-440a-be97-18931f29f144","Type":"ContainerDied","Data":"7a51b07c5581a01862d7637ca943451df22b62d2852ff6cb3fd74d157ffcda3d"} Sep 11 14:34:34.042173 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:34.041977 2553 scope.go:117] "RemoveContainer" containerID="55783a32806c28c56fe94fbefa68fee24dd8a084a0ed2cbb44661b8ec96b645b" Sep 11 14:34:34.057871 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:34.057769 2553 scope.go:117] "RemoveContainer" containerID="55783a32806c28c56fe94fbefa68fee24dd8a084a0ed2cbb44661b8ec96b645b" Sep 11 14:34:34.058213 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:34:34.058172 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55783a32806c28c56fe94fbefa68fee24dd8a084a0ed2cbb44661b8ec96b645b\": container with ID starting with 55783a32806c28c56fe94fbefa68fee24dd8a084a0ed2cbb44661b8ec96b645b not found: ID does not exist" containerID="55783a32806c28c56fe94fbefa68fee24dd8a084a0ed2cbb44661b8ec96b645b" Sep 11 14:34:34.058336 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:34.058213 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55783a32806c28c56fe94fbefa68fee24dd8a084a0ed2cbb44661b8ec96b645b"} err="failed to get container status \"55783a32806c28c56fe94fbefa68fee24dd8a084a0ed2cbb44661b8ec96b645b\": rpc error: code = NotFound desc = could not find container \"55783a32806c28c56fe94fbefa68fee24dd8a084a0ed2cbb44661b8ec96b645b\": container with ID starting with 55783a32806c28c56fe94fbefa68fee24dd8a084a0ed2cbb44661b8ec96b645b not found: ID does not exist" Sep 11 14:34:34.073832 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:34.073757 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx"] Sep 11 14:34:34.077942 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:34.077896 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d7531909-68bd9cc48b-9n7nx"] Sep 11 14:34:34.081776 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:34.081750 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:34.382811 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:34.382703 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:34.712864 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:34.712820 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:34.835325 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:34.835288 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:34.908794 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:34.908744 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:35.340312 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:35.340273 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:35.874830 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:35.874798 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:35.912353 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:35.912317 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14f7e9e8-4592-440a-be97-18931f29f144" path="/var/lib/kubelet/pods/14f7e9e8-4592-440a-be97-18931f29f144/volumes" Sep 11 14:34:36.817883 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:36.817846 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:37.766042 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:37.766009 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6"] Sep 11 14:34:37.766935 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:37.766902 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:37.874833 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:37.874799 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:38.573686 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:38.573646 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:39.006462 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:39.006407 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:40.050831 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.050805 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:40.077112 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.077084 2553 generic.go:356] "Generic (PLEG): container finished" podID="f39c37f6-0d79-49fc-a78a-7f06948d3ef0" containerID="c64bc3915b735c132ce9ffb5457ce9e48f10186ef8129efd1799f84a1946edcd" exitCode=0 Sep 11 14:34:40.077264 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.077132 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" Sep 11 14:34:40.077264 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.077172 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" event={"ID":"f39c37f6-0d79-49fc-a78a-7f06948d3ef0","Type":"ContainerDied","Data":"c64bc3915b735c132ce9ffb5457ce9e48f10186ef8129efd1799f84a1946edcd"} Sep 11 14:34:40.077264 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.077225 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss" event={"ID":"f39c37f6-0d79-49fc-a78a-7f06948d3ef0","Type":"ContainerDied","Data":"a328e0ec5363611c6c47f2673179686054fead975fbe85e1055ed80fe1271f8a"} Sep 11 14:34:40.077264 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.077252 2553 scope.go:117] "RemoveContainer" containerID="c64bc3915b735c132ce9ffb5457ce9e48f10186ef8129efd1799f84a1946edcd" Sep 11 14:34:40.087694 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.087665 2553 scope.go:117] "RemoveContainer" containerID="1693124b05f3dce6908536c81768d951bd53f2256be5b06a9db0af71a35dd35e" Sep 11 14:34:40.096345 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.096260 2553 scope.go:117] "RemoveContainer" containerID="c64bc3915b735c132ce9ffb5457ce9e48f10186ef8129efd1799f84a1946edcd" Sep 11 14:34:40.096704 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:34:40.096682 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c64bc3915b735c132ce9ffb5457ce9e48f10186ef8129efd1799f84a1946edcd\": container with ID starting with c64bc3915b735c132ce9ffb5457ce9e48f10186ef8129efd1799f84a1946edcd not found: ID does not exist" containerID="c64bc3915b735c132ce9ffb5457ce9e48f10186ef8129efd1799f84a1946edcd" Sep 11 14:34:40.096789 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.096715 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c64bc3915b735c132ce9ffb5457ce9e48f10186ef8129efd1799f84a1946edcd"} err="failed to get container status \"c64bc3915b735c132ce9ffb5457ce9e48f10186ef8129efd1799f84a1946edcd\": rpc error: code = NotFound desc = could not find container \"c64bc3915b735c132ce9ffb5457ce9e48f10186ef8129efd1799f84a1946edcd\": container with ID starting with c64bc3915b735c132ce9ffb5457ce9e48f10186ef8129efd1799f84a1946edcd not found: ID does not exist" Sep 11 14:34:40.096789 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.096735 2553 scope.go:117] "RemoveContainer" containerID="1693124b05f3dce6908536c81768d951bd53f2256be5b06a9db0af71a35dd35e" Sep 11 14:34:40.097156 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:34:40.097136 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1693124b05f3dce6908536c81768d951bd53f2256be5b06a9db0af71a35dd35e\": container with ID starting with 1693124b05f3dce6908536c81768d951bd53f2256be5b06a9db0af71a35dd35e not found: ID does not exist" containerID="1693124b05f3dce6908536c81768d951bd53f2256be5b06a9db0af71a35dd35e" Sep 11 14:34:40.097257 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.097159 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1693124b05f3dce6908536c81768d951bd53f2256be5b06a9db0af71a35dd35e"} err="failed to get container status \"1693124b05f3dce6908536c81768d951bd53f2256be5b06a9db0af71a35dd35e\": rpc error: code = NotFound desc = could not find container \"1693124b05f3dce6908536c81768d951bd53f2256be5b06a9db0af71a35dd35e\": container with ID starting with 1693124b05f3dce6908536c81768d951bd53f2256be5b06a9db0af71a35dd35e not found: ID does not exist" Sep 11 14:34:40.140348 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.140315 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istiod-ca-cert\") pod \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " Sep 11 14:34:40.140539 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.140361 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-workload-socket\") pod \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " Sep 11 14:34:40.140539 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.140401 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-envoy\") pod \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " Sep 11 14:34:40.140539 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.140465 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-credential-socket\") pod \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " Sep 11 14:34:40.140539 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.140494 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-podinfo\") pod \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " Sep 11 14:34:40.140539 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.140533 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmpgj\" (UniqueName: \"kubernetes.io/projected/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-kube-api-access-rmpgj\") pod \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " Sep 11 14:34:40.140807 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.140582 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-data\") pod \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " Sep 11 14:34:40.140807 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.140605 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-token\") pod \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " Sep 11 14:34:40.140807 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.140630 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-workload-certs\") pod \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\" (UID: \"f39c37f6-0d79-49fc-a78a-7f06948d3ef0\") " Sep 11 14:34:40.140807 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.140697 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f39c37f6-0d79-49fc-a78a-7f06948d3ef0" (UID: "f39c37f6-0d79-49fc-a78a-7f06948d3ef0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:34:40.141026 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.140805 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f39c37f6-0d79-49fc-a78a-7f06948d3ef0" (UID: "f39c37f6-0d79-49fc-a78a-7f06948d3ef0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:34:40.141026 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.140924 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istiod-ca-cert\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:34:40.141026 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.140940 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-workload-socket\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:34:40.141179 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.141024 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f39c37f6-0d79-49fc-a78a-7f06948d3ef0" (UID: "f39c37f6-0d79-49fc-a78a-7f06948d3ef0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:34:40.141179 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.141051 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f39c37f6-0d79-49fc-a78a-7f06948d3ef0" (UID: "f39c37f6-0d79-49fc-a78a-7f06948d3ef0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:34:40.141179 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.140966 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-data" (OuterVolumeSpecName: "istio-data") pod "f39c37f6-0d79-49fc-a78a-7f06948d3ef0" (UID: "f39c37f6-0d79-49fc-a78a-7f06948d3ef0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:34:40.143009 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.142984 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f39c37f6-0d79-49fc-a78a-7f06948d3ef0" (UID: "f39c37f6-0d79-49fc-a78a-7f06948d3ef0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:34:40.143175 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.143154 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f39c37f6-0d79-49fc-a78a-7f06948d3ef0" (UID: "f39c37f6-0d79-49fc-a78a-7f06948d3ef0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:34:40.143303 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.143276 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-token" (OuterVolumeSpecName: "istio-token") pod "f39c37f6-0d79-49fc-a78a-7f06948d3ef0" (UID: "f39c37f6-0d79-49fc-a78a-7f06948d3ef0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:34:40.143373 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.143286 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-kube-api-access-rmpgj" (OuterVolumeSpecName: "kube-api-access-rmpgj") pod "f39c37f6-0d79-49fc-a78a-7f06948d3ef0" (UID: "f39c37f6-0d79-49fc-a78a-7f06948d3ef0"). InnerVolumeSpecName "kube-api-access-rmpgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:34:40.188599 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.188544 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:40.241304 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.241273 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-data\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:34:40.241304 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.241296 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-token\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:34:40.241304 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.241304 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-workload-certs\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:34:40.241304 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.241313 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-envoy\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:34:40.241596 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.241322 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-credential-socket\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:34:40.241596 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.241330 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-istio-podinfo\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:34:40.241596 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.241339 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmpgj\" (UniqueName: \"kubernetes.io/projected/f39c37f6-0d79-49fc-a78a-7f06948d3ef0-kube-api-access-rmpgj\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:34:40.302034 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.302003 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:40.401824 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.401791 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:40.412071 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.412040 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss"] Sep 11 14:34:40.417716 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:40.417690 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-d7531909-gw-openshift-default-5f7c5c9ccb-5lgss"] Sep 11 14:34:41.061582 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.061543 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:41.176537 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.176501 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:41.293110 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.293065 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5"] Sep 11 14:34:41.293530 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.293515 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14f7e9e8-4592-440a-be97-18931f29f144" containerName="payload-pre-processing" Sep 11 14:34:41.293618 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.293535 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="14f7e9e8-4592-440a-be97-18931f29f144" containerName="payload-pre-processing" Sep 11 14:34:41.293618 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.293556 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f39c37f6-0d79-49fc-a78a-7f06948d3ef0" containerName="istio-proxy" Sep 11 14:34:41.293618 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.293562 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f39c37f6-0d79-49fc-a78a-7f06948d3ef0" containerName="istio-proxy" Sep 11 14:34:41.293618 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.293571 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f39c37f6-0d79-49fc-a78a-7f06948d3ef0" containerName="istio-proxy" Sep 11 14:34:41.293618 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.293576 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f39c37f6-0d79-49fc-a78a-7f06948d3ef0" containerName="istio-proxy" Sep 11 14:34:41.293882 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.293640 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="14f7e9e8-4592-440a-be97-18931f29f144" containerName="payload-pre-processing" Sep 11 14:34:41.293882 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.293649 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f39c37f6-0d79-49fc-a78a-7f06948d3ef0" containerName="istio-proxy" Sep 11 14:34:41.298532 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.298510 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5" Sep 11 14:34:41.303449 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.303429 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c9e2e6b4\"" Sep 11 14:34:41.303726 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.303706 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c9e2e6b4-dockercfg-z5qrl\"" Sep 11 14:34:41.317451 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.317374 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5"] Sep 11 14:34:41.337856 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.337828 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk"] Sep 11 14:34:41.338565 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.338548 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f39c37f6-0d79-49fc-a78a-7f06948d3ef0" containerName="istio-proxy" Sep 11 14:34:41.342050 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.342030 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk" Sep 11 14:34:41.350946 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.350914 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16a97a48-f38b-4108-a38e-75316198b405-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5\" (UID: \"16a97a48-f38b-4108-a38e-75316198b405\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5" Sep 11 14:34:41.351084 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.350991 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt85t\" (UniqueName: \"kubernetes.io/projected/5529f8b9-ba4b-4645-83f9-d8ee059fe44c-kube-api-access-mt85t\") pod \"payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk\" (UID: \"5529f8b9-ba4b-4645-83f9-d8ee059fe44c\") " pod="openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk" Sep 11 14:34:41.351084 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.351040 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hwcf\" (UniqueName: \"kubernetes.io/projected/16a97a48-f38b-4108-a38e-75316198b405-kube-api-access-2hwcf\") pod \"payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5\" (UID: \"16a97a48-f38b-4108-a38e-75316198b405\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5" Sep 11 14:34:41.351084 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.351080 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5529f8b9-ba4b-4645-83f9-d8ee059fe44c-plugins-config-volume\") pod \"payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk\" (UID: \"5529f8b9-ba4b-4645-83f9-d8ee059fe44c\") " pod="openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk" Sep 11 14:34:41.362366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.362331 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk"] Sep 11 14:34:41.456434 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.452744 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16a97a48-f38b-4108-a38e-75316198b405-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5\" (UID: \"16a97a48-f38b-4108-a38e-75316198b405\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5" Sep 11 14:34:41.456434 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.452879 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt85t\" (UniqueName: \"kubernetes.io/projected/5529f8b9-ba4b-4645-83f9-d8ee059fe44c-kube-api-access-mt85t\") pod \"payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk\" (UID: \"5529f8b9-ba4b-4645-83f9-d8ee059fe44c\") " pod="openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk" Sep 11 14:34:41.456434 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.452951 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hwcf\" (UniqueName: \"kubernetes.io/projected/16a97a48-f38b-4108-a38e-75316198b405-kube-api-access-2hwcf\") pod \"payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5\" (UID: \"16a97a48-f38b-4108-a38e-75316198b405\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5" Sep 11 14:34:41.456434 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.453023 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5529f8b9-ba4b-4645-83f9-d8ee059fe44c-plugins-config-volume\") pod \"payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk\" (UID: \"5529f8b9-ba4b-4645-83f9-d8ee059fe44c\") " pod="openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk" Sep 11 14:34:41.456434 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.453855 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16a97a48-f38b-4108-a38e-75316198b405-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5\" (UID: \"16a97a48-f38b-4108-a38e-75316198b405\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5" Sep 11 14:34:41.456434 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.454042 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5529f8b9-ba4b-4645-83f9-d8ee059fe44c-plugins-config-volume\") pod \"payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk\" (UID: \"5529f8b9-ba4b-4645-83f9-d8ee059fe44c\") " pod="openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk" Sep 11 14:34:41.473814 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.473786 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt85t\" (UniqueName: \"kubernetes.io/projected/5529f8b9-ba4b-4645-83f9-d8ee059fe44c-kube-api-access-mt85t\") pod \"payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk\" (UID: \"5529f8b9-ba4b-4645-83f9-d8ee059fe44c\") " pod="openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk" Sep 11 14:34:41.480038 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.480016 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hwcf\" (UniqueName: \"kubernetes.io/projected/16a97a48-f38b-4108-a38e-75316198b405-kube-api-access-2hwcf\") pod \"payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5\" (UID: \"16a97a48-f38b-4108-a38e-75316198b405\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5" Sep 11 14:34:41.609703 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.609612 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5" Sep 11 14:34:41.656020 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.655932 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk" Sep 11 14:34:41.810672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.810633 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:41.912730 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.912700 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f39c37f6-0d79-49fc-a78a-7f06948d3ef0" path="/var/lib/kubelet/pods/f39c37f6-0d79-49fc-a78a-7f06948d3ef0/volumes" Sep 11 14:34:41.917897 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.917875 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5"] Sep 11 14:34:41.920157 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:34:41.920129 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a97a48_f38b_4108_a38e_75316198b405.slice/crio-d203d7d85d1b442ac38fa24fd597967068b6ba67da662529a4076691a6e7a3dc WatchSource:0}: Error finding container d203d7d85d1b442ac38fa24fd597967068b6ba67da662529a4076691a6e7a3dc: Status 404 returned error can't find the container with id d203d7d85d1b442ac38fa24fd597967068b6ba67da662529a4076691a6e7a3dc Sep 11 14:34:41.961168 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:41.961138 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk"] Sep 11 14:34:41.964581 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:34:41.964555 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5529f8b9_ba4b_4645_83f9_d8ee059fe44c.slice/crio-5dd2e86badbbbf5402abe83500085f2b63cbda2666e0bee9aafc4e2eb142ea9f WatchSource:0}: Error finding container 5dd2e86badbbbf5402abe83500085f2b63cbda2666e0bee9aafc4e2eb142ea9f: Status 404 returned error can't find the container with id 5dd2e86badbbbf5402abe83500085f2b63cbda2666e0bee9aafc4e2eb142ea9f Sep 11 14:34:42.093678 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:42.093641 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk" event={"ID":"5529f8b9-ba4b-4645-83f9-d8ee059fe44c","Type":"ContainerStarted","Data":"3821563e147aaf91709b9226dab7e8033bacd50e7ca33128cde09e191c5ff154"} Sep 11 14:34:42.094149 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:42.093709 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk" event={"ID":"5529f8b9-ba4b-4645-83f9-d8ee059fe44c","Type":"ContainerStarted","Data":"5dd2e86badbbbf5402abe83500085f2b63cbda2666e0bee9aafc4e2eb142ea9f"} Sep 11 14:34:42.094149 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:42.093738 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk" Sep 11 14:34:42.095492 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:42.095462 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5" event={"ID":"16a97a48-f38b-4108-a38e-75316198b405","Type":"ContainerStarted","Data":"8e10767ec5740b12bba374ab2cafc78781cf960b828e5a2db23c2c9331bb708a"} Sep 11 14:34:42.095667 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:42.095512 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5" event={"ID":"16a97a48-f38b-4108-a38e-75316198b405","Type":"ContainerStarted","Data":"d203d7d85d1b442ac38fa24fd597967068b6ba67da662529a4076691a6e7a3dc"} Sep 11 14:34:42.095667 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:42.095548 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5" Sep 11 14:34:42.202585 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:42.202492 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk" podStartSLOduration=1.202474385 podStartE2EDuration="1.202474385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:34:42.200902582 +0000 UTC m=+1192.815008927" watchObservedRunningTime="2026-09-11 14:34:42.202474385 +0000 UTC m=+1192.816580729" Sep 11 14:34:42.245358 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:42.245295 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5" podStartSLOduration=1.245273939 podStartE2EDuration="1.245273939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:34:42.244559134 +0000 UTC m=+1192.858665479" watchObservedRunningTime="2026-09-11 14:34:42.245273939 +0000 UTC m=+1192.859380287" Sep 11 14:34:43.408603 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:43.408562 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:44.900672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:44.900644 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:46.197828 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:46.197788 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:48.059457 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:48.059409 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:49.128719 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.128687 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:49.204702 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.204672 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk"] Sep 11 14:34:49.204939 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.204914 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk" podUID="1ee479f4-3ca8-485a-ba93-b6cfa8808775" containerName="payload-processing" containerID="cri-o://268f0354a82f80140536cdfc881a0310a88400d05987e22f3db785b653c4f151" gracePeriod=30 Sep 11 14:34:49.277064 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.277034 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd"] Sep 11 14:34:49.277249 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.277228 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd" podUID="f1c7493d-982c-44d9-b847-b930de23e261" containerName="payload-pre-processing" containerID="cri-o://db118c300959ba8584a449a81a0884b35c2d61bf447291a5be79e86f4dea41fd" gracePeriod=30 Sep 11 14:34:49.297065 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.297039 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:49.436306 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.436225 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:49.530785 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.530758 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk" Sep 11 14:34:49.556230 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.556205 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd" Sep 11 14:34:49.627561 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.627531 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c7493d-982c-44d9-b847-b930de23e261-plugins-config-volume\") pod \"f1c7493d-982c-44d9-b847-b930de23e261\" (UID: \"f1c7493d-982c-44d9-b847-b930de23e261\") " Sep 11 14:34:49.627561 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.627563 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee479f4-3ca8-485a-ba93-b6cfa8808775-plugins-config-volume\") pod \"1ee479f4-3ca8-485a-ba93-b6cfa8808775\" (UID: \"1ee479f4-3ca8-485a-ba93-b6cfa8808775\") " Sep 11 14:34:49.627843 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.627611 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv4tx\" (UniqueName: \"kubernetes.io/projected/1ee479f4-3ca8-485a-ba93-b6cfa8808775-kube-api-access-sv4tx\") pod \"1ee479f4-3ca8-485a-ba93-b6cfa8808775\" (UID: \"1ee479f4-3ca8-485a-ba93-b6cfa8808775\") " Sep 11 14:34:49.627843 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.627631 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjbtm\" (UniqueName: \"kubernetes.io/projected/f1c7493d-982c-44d9-b847-b930de23e261-kube-api-access-gjbtm\") pod \"f1c7493d-982c-44d9-b847-b930de23e261\" (UID: \"f1c7493d-982c-44d9-b847-b930de23e261\") " Sep 11 14:34:49.628025 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.627931 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c7493d-982c-44d9-b847-b930de23e261-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f1c7493d-982c-44d9-b847-b930de23e261" (UID: "f1c7493d-982c-44d9-b847-b930de23e261"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:34:49.628025 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.627959 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ee479f4-3ca8-485a-ba93-b6cfa8808775-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ee479f4-3ca8-485a-ba93-b6cfa8808775" (UID: "1ee479f4-3ca8-485a-ba93-b6cfa8808775"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:34:49.629811 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.629788 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee479f4-3ca8-485a-ba93-b6cfa8808775-kube-api-access-sv4tx" (OuterVolumeSpecName: "kube-api-access-sv4tx") pod "1ee479f4-3ca8-485a-ba93-b6cfa8808775" (UID: "1ee479f4-3ca8-485a-ba93-b6cfa8808775"). InnerVolumeSpecName "kube-api-access-sv4tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:34:49.629811 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.629796 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c7493d-982c-44d9-b847-b930de23e261-kube-api-access-gjbtm" (OuterVolumeSpecName: "kube-api-access-gjbtm") pod "f1c7493d-982c-44d9-b847-b930de23e261" (UID: "f1c7493d-982c-44d9-b847-b930de23e261"). InnerVolumeSpecName "kube-api-access-gjbtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:34:49.665878 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.665836 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:49.728949 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.728909 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1c7493d-982c-44d9-b847-b930de23e261-plugins-config-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:34:49.728949 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.728941 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee479f4-3ca8-485a-ba93-b6cfa8808775-plugins-config-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:34:49.728949 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.728952 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv4tx\" (UniqueName: \"kubernetes.io/projected/1ee479f4-3ca8-485a-ba93-b6cfa8808775-kube-api-access-sv4tx\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:34:49.729193 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.728960 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjbtm\" (UniqueName: \"kubernetes.io/projected/f1c7493d-982c-44d9-b847-b930de23e261-kube-api-access-gjbtm\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:34:49.765404 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.765365 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:49.917614 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.917528 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmqjh_6b2452d9-1730-458c-b380-dbcfd9e5be19/ovn-acl-logging/0.log" Sep 11 14:34:49.920803 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:49.920575 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmqjh_6b2452d9-1730-458c-b380-dbcfd9e5be19/ovn-acl-logging/0.log" Sep 11 14:34:50.136662 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:50.136622 2553 generic.go:356] "Generic (PLEG): container finished" podID="f1c7493d-982c-44d9-b847-b930de23e261" containerID="db118c300959ba8584a449a81a0884b35c2d61bf447291a5be79e86f4dea41fd" exitCode=0 Sep 11 14:34:50.137095 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:50.136676 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd" Sep 11 14:34:50.137095 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:50.136703 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd" event={"ID":"f1c7493d-982c-44d9-b847-b930de23e261","Type":"ContainerDied","Data":"db118c300959ba8584a449a81a0884b35c2d61bf447291a5be79e86f4dea41fd"} Sep 11 14:34:50.137095 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:50.136760 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd" event={"ID":"f1c7493d-982c-44d9-b847-b930de23e261","Type":"ContainerDied","Data":"5934553c981eb0dbaaaf8360aa645659dc01549d6896722c4cc0b940c5e7772e"} Sep 11 14:34:50.137095 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:50.136793 2553 scope.go:117] "RemoveContainer" containerID="db118c300959ba8584a449a81a0884b35c2d61bf447291a5be79e86f4dea41fd" Sep 11 14:34:50.138176 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:50.138153 2553 generic.go:356] "Generic (PLEG): container finished" podID="1ee479f4-3ca8-485a-ba93-b6cfa8808775" containerID="268f0354a82f80140536cdfc881a0310a88400d05987e22f3db785b653c4f151" exitCode=0 Sep 11 14:34:50.138305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:50.138206 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk" Sep 11 14:34:50.138305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:50.138209 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk" event={"ID":"1ee479f4-3ca8-485a-ba93-b6cfa8808775","Type":"ContainerDied","Data":"268f0354a82f80140536cdfc881a0310a88400d05987e22f3db785b653c4f151"} Sep 11 14:34:50.138305 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:50.138244 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk" event={"ID":"1ee479f4-3ca8-485a-ba93-b6cfa8808775","Type":"ContainerDied","Data":"68add7603767fe2f1ffc0f648739b82ae69830ad224e3c9bd92a413d2a1959d6"} Sep 11 14:34:50.145213 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:50.145197 2553 scope.go:117] "RemoveContainer" containerID="db118c300959ba8584a449a81a0884b35c2d61bf447291a5be79e86f4dea41fd" Sep 11 14:34:50.145519 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:34:50.145498 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db118c300959ba8584a449a81a0884b35c2d61bf447291a5be79e86f4dea41fd\": container with ID starting with db118c300959ba8584a449a81a0884b35c2d61bf447291a5be79e86f4dea41fd not found: ID does not exist" containerID="db118c300959ba8584a449a81a0884b35c2d61bf447291a5be79e86f4dea41fd" Sep 11 14:34:50.145601 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:50.145526 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db118c300959ba8584a449a81a0884b35c2d61bf447291a5be79e86f4dea41fd"} err="failed to get container status \"db118c300959ba8584a449a81a0884b35c2d61bf447291a5be79e86f4dea41fd\": rpc error: code = NotFound desc = could not find container \"db118c300959ba8584a449a81a0884b35c2d61bf447291a5be79e86f4dea41fd\": container with ID starting with db118c300959ba8584a449a81a0884b35c2d61bf447291a5be79e86f4dea41fd not found: ID does not exist" Sep 11 14:34:50.145601 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:50.145544 2553 scope.go:117] "RemoveContainer" containerID="268f0354a82f80140536cdfc881a0310a88400d05987e22f3db785b653c4f151" Sep 11 14:34:50.151537 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:50.151516 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:50.153369 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:50.153356 2553 scope.go:117] "RemoveContainer" containerID="268f0354a82f80140536cdfc881a0310a88400d05987e22f3db785b653c4f151" Sep 11 14:34:50.153656 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:34:50.153638 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"268f0354a82f80140536cdfc881a0310a88400d05987e22f3db785b653c4f151\": container with ID starting with 268f0354a82f80140536cdfc881a0310a88400d05987e22f3db785b653c4f151 not found: ID does not exist" containerID="268f0354a82f80140536cdfc881a0310a88400d05987e22f3db785b653c4f151" Sep 11 14:34:50.153711 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:50.153670 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"268f0354a82f80140536cdfc881a0310a88400d05987e22f3db785b653c4f151"} err="failed to get container status \"268f0354a82f80140536cdfc881a0310a88400d05987e22f3db785b653c4f151\": rpc error: code = NotFound desc = could not find container \"268f0354a82f80140536cdfc881a0310a88400d05987e22f3db785b653c4f151\": container with ID starting with 268f0354a82f80140536cdfc881a0310a88400d05987e22f3db785b653c4f151 not found: ID does not exist" Sep 11 14:34:50.189541 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:50.189475 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk"] Sep 11 14:34:50.199099 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:50.199076 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c322a2-57d754d867-bgffk"] Sep 11 14:34:50.216022 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:50.215986 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd"] Sep 11 14:34:50.225073 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:50.225048 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c322a2-7c47dd94db-zshcd"] Sep 11 14:34:50.292995 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:50.292963 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:51.688064 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:51.688010 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:51.914150 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:51.914114 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ee479f4-3ca8-485a-ba93-b6cfa8808775" path="/var/lib/kubelet/pods/1ee479f4-3ca8-485a-ba93-b6cfa8808775/volumes" Sep 11 14:34:51.914680 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:51.914662 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1c7493d-982c-44d9-b847-b930de23e261" path="/var/lib/kubelet/pods/f1c7493d-982c-44d9-b847-b930de23e261/volumes" Sep 11 14:34:52.785811 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:52.785770 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" podUID="f6a01974-56cc-45fa-b284-922c51bbf554" containerName="main" containerID="cri-o://551d8e1f96714abe123e5c3d4deb97dac5d5af70c82b88d5abc596e63da5a25d" gracePeriod=45 Sep 11 14:34:52.830943 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:52.830908 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:53.042178 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.042119 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:53.102223 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.102197 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5" Sep 11 14:34:53.102380 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.102348 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk" Sep 11 14:34:53.156117 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.156088 2553 generic.go:356] "Generic (PLEG): container finished" podID="f6a01974-56cc-45fa-b284-922c51bbf554" containerID="551d8e1f96714abe123e5c3d4deb97dac5d5af70c82b88d5abc596e63da5a25d" exitCode=0 Sep 11 14:34:53.156293 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.156141 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" Sep 11 14:34:53.156293 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.156174 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" event={"ID":"f6a01974-56cc-45fa-b284-922c51bbf554","Type":"ContainerDied","Data":"551d8e1f96714abe123e5c3d4deb97dac5d5af70c82b88d5abc596e63da5a25d"} Sep 11 14:34:53.156293 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.156226 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6" event={"ID":"f6a01974-56cc-45fa-b284-922c51bbf554","Type":"ContainerDied","Data":"a4a6ce85c5934a6ddb222f995e841e726ac5871a7bd3773603e5db6e369acce9"} Sep 11 14:34:53.156293 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.156256 2553 scope.go:117] "RemoveContainer" containerID="551d8e1f96714abe123e5c3d4deb97dac5d5af70c82b88d5abc596e63da5a25d" Sep 11 14:34:53.159771 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.159746 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-dshm\") pod \"f6a01974-56cc-45fa-b284-922c51bbf554\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " Sep 11 14:34:53.159904 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.159807 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a01974-56cc-45fa-b284-922c51bbf554-tls-certs\") pod \"f6a01974-56cc-45fa-b284-922c51bbf554\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " Sep 11 14:34:53.159904 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.159858 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-model-cache\") pod \"f6a01974-56cc-45fa-b284-922c51bbf554\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " Sep 11 14:34:53.160025 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.159945 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgq42\" (UniqueName: \"kubernetes.io/projected/f6a01974-56cc-45fa-b284-922c51bbf554-kube-api-access-fgq42\") pod \"f6a01974-56cc-45fa-b284-922c51bbf554\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " Sep 11 14:34:53.160025 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.159987 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-home\") pod \"f6a01974-56cc-45fa-b284-922c51bbf554\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " Sep 11 14:34:53.160132 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.160031 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-tmp-dir\") pod \"f6a01974-56cc-45fa-b284-922c51bbf554\" (UID: \"f6a01974-56cc-45fa-b284-922c51bbf554\") " Sep 11 14:34:53.160439 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.160242 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6a01974-56cc-45fa-b284-922c51bbf554" (UID: "f6a01974-56cc-45fa-b284-922c51bbf554"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:34:53.160525 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.160482 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-home" (OuterVolumeSpecName: "home") pod "f6a01974-56cc-45fa-b284-922c51bbf554" (UID: "f6a01974-56cc-45fa-b284-922c51bbf554"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:34:53.160798 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.160721 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6a01974-56cc-45fa-b284-922c51bbf554" (UID: "f6a01974-56cc-45fa-b284-922c51bbf554"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:34:53.161009 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.160989 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-home\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:34:53.161101 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.161014 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-tmp-dir\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:34:53.161101 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.161027 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-model-cache\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:34:53.163083 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.163056 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a01974-56cc-45fa-b284-922c51bbf554-kube-api-access-fgq42" (OuterVolumeSpecName: "kube-api-access-fgq42") pod "f6a01974-56cc-45fa-b284-922c51bbf554" (UID: "f6a01974-56cc-45fa-b284-922c51bbf554"). InnerVolumeSpecName "kube-api-access-fgq42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:34:53.163817 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.163793 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a01974-56cc-45fa-b284-922c51bbf554-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6a01974-56cc-45fa-b284-922c51bbf554" (UID: "f6a01974-56cc-45fa-b284-922c51bbf554"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:34:53.163914 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.163883 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-dshm" (OuterVolumeSpecName: "dshm") pod "f6a01974-56cc-45fa-b284-922c51bbf554" (UID: "f6a01974-56cc-45fa-b284-922c51bbf554"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:34:53.167481 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.167458 2553 scope.go:117] "RemoveContainer" containerID="551d8e1f96714abe123e5c3d4deb97dac5d5af70c82b88d5abc596e63da5a25d" Sep 11 14:34:53.167815 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:34:53.167795 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"551d8e1f96714abe123e5c3d4deb97dac5d5af70c82b88d5abc596e63da5a25d\": container with ID starting with 551d8e1f96714abe123e5c3d4deb97dac5d5af70c82b88d5abc596e63da5a25d not found: ID does not exist" containerID="551d8e1f96714abe123e5c3d4deb97dac5d5af70c82b88d5abc596e63da5a25d" Sep 11 14:34:53.167905 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.167821 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"551d8e1f96714abe123e5c3d4deb97dac5d5af70c82b88d5abc596e63da5a25d"} err="failed to get container status \"551d8e1f96714abe123e5c3d4deb97dac5d5af70c82b88d5abc596e63da5a25d\": rpc error: code = NotFound desc = could not find container \"551d8e1f96714abe123e5c3d4deb97dac5d5af70c82b88d5abc596e63da5a25d\": container with ID starting with 551d8e1f96714abe123e5c3d4deb97dac5d5af70c82b88d5abc596e63da5a25d not found: ID does not exist" Sep 11 14:34:53.262375 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.262345 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a01974-56cc-45fa-b284-922c51bbf554-dshm\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:34:53.262375 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.262372 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a01974-56cc-45fa-b284-922c51bbf554-tls-certs\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:34:53.262375 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.262383 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgq42\" (UniqueName: \"kubernetes.io/projected/f6a01974-56cc-45fa-b284-922c51bbf554-kube-api-access-fgq42\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:34:53.493784 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.493754 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6"] Sep 11 14:34:53.498322 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.498294 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-c322a2/autores-model-c322a2-kserve-686b9976c5-45tk6"] Sep 11 14:34:53.915438 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:53.914207 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6a01974-56cc-45fa-b284-922c51bbf554" path="/var/lib/kubelet/pods/f6a01974-56cc-45fa-b284-922c51bbf554/volumes" Sep 11 14:34:54.304335 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:54.304246 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:55.642021 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:55.641984 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:56.897282 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:56.897247 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:58.012617 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:58.012580 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:58.246331 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:58.246295 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9e2e6b4-8glgr"] Sep 11 14:34:58.246775 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:58.246757 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1c7493d-982c-44d9-b847-b930de23e261" containerName="payload-pre-processing" Sep 11 14:34:58.246872 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:58.246778 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1c7493d-982c-44d9-b847-b930de23e261" containerName="payload-pre-processing" Sep 11 14:34:58.246872 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:58.246789 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ee479f4-3ca8-485a-ba93-b6cfa8808775" containerName="payload-processing" Sep 11 14:34:58.246872 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:58.246797 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ee479f4-3ca8-485a-ba93-b6cfa8808775" containerName="payload-processing" Sep 11 14:34:58.246872 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:58.246813 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6a01974-56cc-45fa-b284-922c51bbf554" containerName="main" Sep 11 14:34:58.246872 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:58.246821 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6a01974-56cc-45fa-b284-922c51bbf554" containerName="main" Sep 11 14:34:58.247110 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:58.246939 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1c7493d-982c-44d9-b847-b930de23e261" containerName="payload-pre-processing" Sep 11 14:34:58.247110 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:58.246956 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ee479f4-3ca8-485a-ba93-b6cfa8808775" containerName="payload-processing" Sep 11 14:34:58.247110 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:58.246965 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6a01974-56cc-45fa-b284-922c51bbf554" containerName="main" Sep 11 14:34:58.250950 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:58.250929 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9e2e6b4-8glgr" Sep 11 14:34:58.254427 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:58.254400 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:34:58.254427 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:58.254408 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6975v\"" Sep 11 14:34:58.259898 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:58.259874 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9e2e6b4-8glgr"] Sep 11 14:34:58.308047 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:58.307935 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/852601cc-c41f-4043-95a6-7493ffd21f32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c9e2e6b4-8glgr\" (UID: \"852601cc-c41f-4043-95a6-7493ffd21f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9e2e6b4-8glgr" Sep 11 14:34:58.409157 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:58.409115 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/852601cc-c41f-4043-95a6-7493ffd21f32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c9e2e6b4-8glgr\" (UID: \"852601cc-c41f-4043-95a6-7493ffd21f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9e2e6b4-8glgr" Sep 11 14:34:58.409787 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:58.409762 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/852601cc-c41f-4043-95a6-7493ffd21f32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c9e2e6b4-8glgr\" (UID: \"852601cc-c41f-4043-95a6-7493ffd21f32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9e2e6b4-8glgr" Sep 11 14:34:58.563363 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:58.563269 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9e2e6b4-8glgr" Sep 11 14:34:58.845387 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:58.845365 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9e2e6b4-8glgr"] Sep 11 14:34:58.847388 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:34:58.847355 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852601cc_c41f_4043_95a6_7493ffd21f32.slice/crio-1839be1838bf102b9dfe7ad29586b31a4f127c73e20e9bb3c1c1eaa3c98ca224 WatchSource:0}: Error finding container 1839be1838bf102b9dfe7ad29586b31a4f127c73e20e9bb3c1c1eaa3c98ca224: Status 404 returned error can't find the container with id 1839be1838bf102b9dfe7ad29586b31a4f127c73e20e9bb3c1c1eaa3c98ca224 Sep 11 14:34:58.877120 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:58.877083 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:59.081087 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:59.081046 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:59.190240 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:59.190206 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9e2e6b4-8glgr" event={"ID":"852601cc-c41f-4043-95a6-7493ffd21f32","Type":"ContainerStarted","Data":"1839be1838bf102b9dfe7ad29586b31a4f127c73e20e9bb3c1c1eaa3c98ca224"} Sep 11 14:34:59.269888 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:59.269857 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:59.506393 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:59.506305 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:34:59.766134 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:34:59.766046 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:00.103916 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.103832 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j"] Sep 11 14:35:00.108103 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.108080 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.111599 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.111578 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 14:35:00.111708 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.111620 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-69a7cf-openshift-default-dockercfg-j6cqq\"" Sep 11 14:35:00.126964 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.126780 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j"] Sep 11 14:35:00.132521 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.132387 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:00.226477 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.226439 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e92780-d597-45c4-824d-9b453e24d070-istiod-ca-cert\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.226682 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.226503 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e92780-d597-45c4-824d-9b453e24d070-istio-token\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.226682 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.226564 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-workload-socket\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.226682 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.226621 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e92780-d597-45c4-824d-9b453e24d070-istio-podinfo\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.226682 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.226669 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-istio-envoy\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.226902 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.226699 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-workload-certs\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.226902 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.226770 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-credential-socket\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.226902 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.226847 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p88c7\" (UniqueName: \"kubernetes.io/projected/e5e92780-d597-45c4-824d-9b453e24d070-kube-api-access-p88c7\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.226902 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.226884 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-istio-data\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.327634 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.327591 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-workload-certs\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.327826 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.327685 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-credential-socket\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.327826 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.327729 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p88c7\" (UniqueName: \"kubernetes.io/projected/e5e92780-d597-45c4-824d-9b453e24d070-kube-api-access-p88c7\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.327826 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.327762 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-istio-data\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.328004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.327855 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e92780-d597-45c4-824d-9b453e24d070-istiod-ca-cert\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.328004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.327901 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e92780-d597-45c4-824d-9b453e24d070-istio-token\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.328004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.327984 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-workload-socket\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.328286 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.328023 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e92780-d597-45c4-824d-9b453e24d070-istio-podinfo\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.328286 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.328031 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-workload-certs\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.328286 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.328063 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-istio-envoy\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.328680 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.328651 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-workload-socket\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.328855 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.328834 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e92780-d597-45c4-824d-9b453e24d070-istiod-ca-cert\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.329092 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.329068 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-credential-socket\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.329442 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.329394 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-istio-data\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.330971 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.330766 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-istio-envoy\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.331698 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.331680 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e92780-d597-45c4-824d-9b453e24d070-istio-podinfo\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.340002 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.339971 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e92780-d597-45c4-824d-9b453e24d070-istio-token\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.341358 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.341337 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p88c7\" (UniqueName: \"kubernetes.io/projected/e5e92780-d597-45c4-824d-9b453e24d070-kube-api-access-p88c7\") pod \"e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.421494 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.421458 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:00.665832 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.665794 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:00.727516 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.727379 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j"] Sep 11 14:35:00.785540 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:00.785466 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:01.229735 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:35:01.229695 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e92780_d597_45c4_824d_9b453e24d070.slice/crio-1f99dff15a154528caa11afc967b21c4049bb6aa10d65a485b819e91e3c65657 WatchSource:0}: Error finding container 1f99dff15a154528caa11afc967b21c4049bb6aa10d65a485b819e91e3c65657: Status 404 returned error can't find the container with id 1f99dff15a154528caa11afc967b21c4049bb6aa10d65a485b819e91e3c65657 Sep 11 14:35:01.232260 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:01.232224 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:35:01.232347 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:01.232300 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:35:01.232347 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:01.232331 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:35:01.949697 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:01.949640 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:02.209100 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:02.209011 2553 generic.go:356] "Generic (PLEG): container finished" podID="852601cc-c41f-4043-95a6-7493ffd21f32" containerID="b46bed8f188f97e6e07301e68967690b7ece03e9a140177c576ccdc5c81ef070" exitCode=0 Sep 11 14:35:02.209100 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:02.209088 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9e2e6b4-8glgr" event={"ID":"852601cc-c41f-4043-95a6-7493ffd21f32","Type":"ContainerDied","Data":"b46bed8f188f97e6e07301e68967690b7ece03e9a140177c576ccdc5c81ef070"} Sep 11 14:35:02.210615 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:02.210589 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" event={"ID":"e5e92780-d597-45c4-824d-9b453e24d070","Type":"ContainerStarted","Data":"55b5aa39b15b2800054928f630bf2f4ac0079701ed3187ca21f47858cca489c8"} Sep 11 14:35:02.210739 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:02.210632 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" event={"ID":"e5e92780-d597-45c4-824d-9b453e24d070","Type":"ContainerStarted","Data":"1f99dff15a154528caa11afc967b21c4049bb6aa10d65a485b819e91e3c65657"} Sep 11 14:35:02.331122 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:02.331073 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" podStartSLOduration=2.331055328 podStartE2EDuration="2.331055328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:35:02.329829305 +0000 UTC m=+1212.943935682" watchObservedRunningTime="2026-09-11 14:35:02.331055328 +0000 UTC m=+1212.945161705" Sep 11 14:35:02.422116 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:02.422084 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:02.427444 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:02.427404 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:03.215940 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:03.215910 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:03.217306 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:03.217286 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:03.218888 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:03.218866 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:03.357688 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:03.357663 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9e2e6b4-8glgr" Sep 11 14:35:03.456195 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:03.456159 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/852601cc-c41f-4043-95a6-7493ffd21f32-maas-api-service-ca\") pod \"852601cc-c41f-4043-95a6-7493ffd21f32\" (UID: \"852601cc-c41f-4043-95a6-7493ffd21f32\") " Sep 11 14:35:03.456651 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:03.456624 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852601cc-c41f-4043-95a6-7493ffd21f32-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "852601cc-c41f-4043-95a6-7493ffd21f32" (UID: "852601cc-c41f-4043-95a6-7493ffd21f32"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:35:03.557144 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:03.557067 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/852601cc-c41f-4043-95a6-7493ffd21f32-maas-api-service-ca\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:35:04.221719 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:04.221617 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9e2e6b4-8glgr" event={"ID":"852601cc-c41f-4043-95a6-7493ffd21f32","Type":"ContainerDied","Data":"1839be1838bf102b9dfe7ad29586b31a4f127c73e20e9bb3c1c1eaa3c98ca224"} Sep 11 14:35:04.221719 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:04.221702 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1839be1838bf102b9dfe7ad29586b31a4f127c73e20e9bb3c1c1eaa3c98ca224" Sep 11 14:35:04.221719 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:04.221639 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9e2e6b4-8glgr" Sep 11 14:35:04.515316 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:04.515224 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:04.658560 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:04.658528 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:04.802882 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:04.802785 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:05.977580 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:05.977534 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:06.081975 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:06.081939 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:06.174469 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:06.174437 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:06.737039 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:06.737001 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:06.866975 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:06.866933 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:07.750485 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:07.750450 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:08.836834 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:08.836796 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:09.829328 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:09.829283 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:11.492637 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:11.492608 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:12.587005 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:12.586965 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:12.652157 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:12.652125 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk"] Sep 11 14:35:12.652681 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:12.652619 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk" podUID="5529f8b9-ba4b-4645-83f9-d8ee059fe44c" containerName="payload-processing" containerID="cri-o://3821563e147aaf91709b9226dab7e8033bacd50e7ca33128cde09e191c5ff154" gracePeriod=30 Sep 11 14:35:12.666465 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:12.666438 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5"] Sep 11 14:35:12.666695 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:12.666672 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5" podUID="16a97a48-f38b-4108-a38e-75316198b405" containerName="payload-pre-processing" containerID="cri-o://8e10767ec5740b12bba374ab2cafc78781cf960b828e5a2db23c2c9331bb708a" gracePeriod=30 Sep 11 14:35:12.692979 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:12.692943 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:12.798561 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:12.798510 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:12.914599 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:12.914545 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:12.923579 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:12.923556 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk" Sep 11 14:35:12.959810 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:12.959775 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5" Sep 11 14:35:12.990284 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:12.990241 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9e2e6b4-8glgr"] Sep 11 14:35:13.008920 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.008882 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:13.010612 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.010585 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c9e2e6b4-8glgr"] Sep 11 14:35:13.051583 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.051550 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mt85t\" (UniqueName: \"kubernetes.io/projected/5529f8b9-ba4b-4645-83f9-d8ee059fe44c-kube-api-access-mt85t\") pod \"5529f8b9-ba4b-4645-83f9-d8ee059fe44c\" (UID: \"5529f8b9-ba4b-4645-83f9-d8ee059fe44c\") " Sep 11 14:35:13.051765 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.051592 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5529f8b9-ba4b-4645-83f9-d8ee059fe44c-plugins-config-volume\") pod \"5529f8b9-ba4b-4645-83f9-d8ee059fe44c\" (UID: \"5529f8b9-ba4b-4645-83f9-d8ee059fe44c\") " Sep 11 14:35:13.051836 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.051752 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hwcf\" (UniqueName: \"kubernetes.io/projected/16a97a48-f38b-4108-a38e-75316198b405-kube-api-access-2hwcf\") pod \"16a97a48-f38b-4108-a38e-75316198b405\" (UID: \"16a97a48-f38b-4108-a38e-75316198b405\") " Sep 11 14:35:13.051836 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.051815 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16a97a48-f38b-4108-a38e-75316198b405-plugins-config-volume\") pod \"16a97a48-f38b-4108-a38e-75316198b405\" (UID: \"16a97a48-f38b-4108-a38e-75316198b405\") " Sep 11 14:35:13.052167 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.052061 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5529f8b9-ba4b-4645-83f9-d8ee059fe44c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5529f8b9-ba4b-4645-83f9-d8ee059fe44c" (UID: "5529f8b9-ba4b-4645-83f9-d8ee059fe44c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:35:13.052292 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.052244 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16a97a48-f38b-4108-a38e-75316198b405-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "16a97a48-f38b-4108-a38e-75316198b405" (UID: "16a97a48-f38b-4108-a38e-75316198b405"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:35:13.052379 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.052340 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5529f8b9-ba4b-4645-83f9-d8ee059fe44c-plugins-config-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:35:13.052461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.052387 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16a97a48-f38b-4108-a38e-75316198b405-plugins-config-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:35:13.054245 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.054214 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5529f8b9-ba4b-4645-83f9-d8ee059fe44c-kube-api-access-mt85t" (OuterVolumeSpecName: "kube-api-access-mt85t") pod "5529f8b9-ba4b-4645-83f9-d8ee059fe44c" (UID: "5529f8b9-ba4b-4645-83f9-d8ee059fe44c"). InnerVolumeSpecName "kube-api-access-mt85t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:35:13.054478 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.054451 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a97a48-f38b-4108-a38e-75316198b405-kube-api-access-2hwcf" (OuterVolumeSpecName: "kube-api-access-2hwcf") pod "16a97a48-f38b-4108-a38e-75316198b405" (UID: "16a97a48-f38b-4108-a38e-75316198b405"). InnerVolumeSpecName "kube-api-access-2hwcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:35:13.153355 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.153326 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt85t\" (UniqueName: \"kubernetes.io/projected/5529f8b9-ba4b-4645-83f9-d8ee059fe44c-kube-api-access-mt85t\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:35:13.153355 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.153354 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hwcf\" (UniqueName: \"kubernetes.io/projected/16a97a48-f38b-4108-a38e-75316198b405-kube-api-access-2hwcf\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:35:13.271249 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.271216 2553 generic.go:356] "Generic (PLEG): container finished" podID="5529f8b9-ba4b-4645-83f9-d8ee059fe44c" containerID="3821563e147aaf91709b9226dab7e8033bacd50e7ca33128cde09e191c5ff154" exitCode=0 Sep 11 14:35:13.271482 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.271280 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk" Sep 11 14:35:13.271482 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.271298 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk" event={"ID":"5529f8b9-ba4b-4645-83f9-d8ee059fe44c","Type":"ContainerDied","Data":"3821563e147aaf91709b9226dab7e8033bacd50e7ca33128cde09e191c5ff154"} Sep 11 14:35:13.271482 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.271399 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk" event={"ID":"5529f8b9-ba4b-4645-83f9-d8ee059fe44c","Type":"ContainerDied","Data":"5dd2e86badbbbf5402abe83500085f2b63cbda2666e0bee9aafc4e2eb142ea9f"} Sep 11 14:35:13.271482 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.271456 2553 scope.go:117] "RemoveContainer" containerID="3821563e147aaf91709b9226dab7e8033bacd50e7ca33128cde09e191c5ff154" Sep 11 14:35:13.272839 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.272816 2553 generic.go:356] "Generic (PLEG): container finished" podID="16a97a48-f38b-4108-a38e-75316198b405" containerID="8e10767ec5740b12bba374ab2cafc78781cf960b828e5a2db23c2c9331bb708a" exitCode=0 Sep 11 14:35:13.272917 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.272871 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5" Sep 11 14:35:13.272917 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.272901 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5" event={"ID":"16a97a48-f38b-4108-a38e-75316198b405","Type":"ContainerDied","Data":"8e10767ec5740b12bba374ab2cafc78781cf960b828e5a2db23c2c9331bb708a"} Sep 11 14:35:13.273000 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.272949 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5" event={"ID":"16a97a48-f38b-4108-a38e-75316198b405","Type":"ContainerDied","Data":"d203d7d85d1b442ac38fa24fd597967068b6ba67da662529a4076691a6e7a3dc"} Sep 11 14:35:13.280235 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.280120 2553 scope.go:117] "RemoveContainer" containerID="3821563e147aaf91709b9226dab7e8033bacd50e7ca33128cde09e191c5ff154" Sep 11 14:35:13.280536 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:35:13.280499 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3821563e147aaf91709b9226dab7e8033bacd50e7ca33128cde09e191c5ff154\": container with ID starting with 3821563e147aaf91709b9226dab7e8033bacd50e7ca33128cde09e191c5ff154 not found: ID does not exist" containerID="3821563e147aaf91709b9226dab7e8033bacd50e7ca33128cde09e191c5ff154" Sep 11 14:35:13.280627 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.280546 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3821563e147aaf91709b9226dab7e8033bacd50e7ca33128cde09e191c5ff154"} err="failed to get container status \"3821563e147aaf91709b9226dab7e8033bacd50e7ca33128cde09e191c5ff154\": rpc error: code = NotFound desc = could not find container \"3821563e147aaf91709b9226dab7e8033bacd50e7ca33128cde09e191c5ff154\": container with ID starting with 3821563e147aaf91709b9226dab7e8033bacd50e7ca33128cde09e191c5ff154 not found: ID does not exist" Sep 11 14:35:13.280627 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.280568 2553 scope.go:117] "RemoveContainer" containerID="8e10767ec5740b12bba374ab2cafc78781cf960b828e5a2db23c2c9331bb708a" Sep 11 14:35:13.288786 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.288768 2553 scope.go:117] "RemoveContainer" containerID="8e10767ec5740b12bba374ab2cafc78781cf960b828e5a2db23c2c9331bb708a" Sep 11 14:35:13.289032 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:35:13.289013 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e10767ec5740b12bba374ab2cafc78781cf960b828e5a2db23c2c9331bb708a\": container with ID starting with 8e10767ec5740b12bba374ab2cafc78781cf960b828e5a2db23c2c9331bb708a not found: ID does not exist" containerID="8e10767ec5740b12bba374ab2cafc78781cf960b828e5a2db23c2c9331bb708a" Sep 11 14:35:13.289090 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.289044 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e10767ec5740b12bba374ab2cafc78781cf960b828e5a2db23c2c9331bb708a"} err="failed to get container status \"8e10767ec5740b12bba374ab2cafc78781cf960b828e5a2db23c2c9331bb708a\": rpc error: code = NotFound desc = could not find container \"8e10767ec5740b12bba374ab2cafc78781cf960b828e5a2db23c2c9331bb708a\": container with ID starting with 8e10767ec5740b12bba374ab2cafc78781cf960b828e5a2db23c2c9331bb708a not found: ID does not exist" Sep 11 14:35:13.305771 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.305743 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5"] Sep 11 14:35:13.307523 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.307500 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c9e2e6b4-77cd858c46-w7zp5"] Sep 11 14:35:13.322228 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.322208 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk"] Sep 11 14:35:13.326391 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.326372 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9e2e6b4-769b68747f-v9xrk"] Sep 11 14:35:13.480469 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.480352 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:13.911790 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.911759 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16a97a48-f38b-4108-a38e-75316198b405" path="/var/lib/kubelet/pods/16a97a48-f38b-4108-a38e-75316198b405/volumes" Sep 11 14:35:13.912167 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.912131 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5529f8b9-ba4b-4645-83f9-d8ee059fe44c" path="/var/lib/kubelet/pods/5529f8b9-ba4b-4645-83f9-d8ee059fe44c/volumes" Sep 11 14:35:13.912442 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:13.912429 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="852601cc-c41f-4043-95a6-7493ffd21f32" path="/var/lib/kubelet/pods/852601cc-c41f-4043-95a6-7493ffd21f32/volumes" Sep 11 14:35:14.254167 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:14.254061 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:14.856122 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:14.856082 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:15.293624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.293591 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:15.371936 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.371902 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd"] Sep 11 14:35:15.372530 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.372505 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16a97a48-f38b-4108-a38e-75316198b405" containerName="payload-pre-processing" Sep 11 14:35:15.372530 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.372529 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="16a97a48-f38b-4108-a38e-75316198b405" containerName="payload-pre-processing" Sep 11 14:35:15.372720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.372547 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="852601cc-c41f-4043-95a6-7493ffd21f32" containerName="revoke-keys" Sep 11 14:35:15.372720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.372555 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="852601cc-c41f-4043-95a6-7493ffd21f32" containerName="revoke-keys" Sep 11 14:35:15.372720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.372568 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5529f8b9-ba4b-4645-83f9-d8ee059fe44c" containerName="payload-processing" Sep 11 14:35:15.372720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.372576 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5529f8b9-ba4b-4645-83f9-d8ee059fe44c" containerName="payload-processing" Sep 11 14:35:15.372720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.372686 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="852601cc-c41f-4043-95a6-7493ffd21f32" containerName="revoke-keys" Sep 11 14:35:15.372720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.372698 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5529f8b9-ba4b-4645-83f9-d8ee059fe44c" containerName="payload-processing" Sep 11 14:35:15.372720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.372708 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="16a97a48-f38b-4108-a38e-75316198b405" containerName="payload-pre-processing" Sep 11 14:35:15.377592 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.377571 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.380642 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.380588 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-15594dcc-openshift-default-dockercfg-d4j4l\"" Sep 11 14:35:15.403530 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.403505 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd"] Sep 11 14:35:15.477847 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.477813 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-workload-socket\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.477847 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.477850 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-credential-socket\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.478089 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.477868 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-token\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.478089 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.477913 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9740e363-d91f-4efd-b5a6-f487ec62151c-istiod-ca-cert\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.478089 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.477977 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-data\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.478089 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.477993 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-podinfo\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.478089 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.478016 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-envoy\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.478302 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.478086 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-workload-certs\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.478302 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.478127 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p24pm\" (UniqueName: \"kubernetes.io/projected/9740e363-d91f-4efd-b5a6-f487ec62151c-kube-api-access-p24pm\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.480172 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.480144 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:15.578854 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.578762 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9740e363-d91f-4efd-b5a6-f487ec62151c-istiod-ca-cert\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.578854 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.578839 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-data\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.579070 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.578863 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-podinfo\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.579070 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.578886 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-envoy\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.579070 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.578914 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-workload-certs\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.579070 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.578941 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p24pm\" (UniqueName: \"kubernetes.io/projected/9740e363-d91f-4efd-b5a6-f487ec62151c-kube-api-access-p24pm\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.579070 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.578975 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-workload-socket\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.579070 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.579006 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-credential-socket\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.579070 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.579027 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-token\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.579654 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.579236 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-data\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.579654 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.579566 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-workload-socket\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.579654 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.579575 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9740e363-d91f-4efd-b5a6-f487ec62151c-istiod-ca-cert\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.579815 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.579650 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-credential-socket\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.579815 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.579698 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-workload-certs\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.581319 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.581299 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-envoy\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.581803 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.581783 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-podinfo\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.611719 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.611683 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-token\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.611910 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.611895 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p24pm\" (UniqueName: \"kubernetes.io/projected/9740e363-d91f-4efd-b5a6-f487ec62151c-kube-api-access-p24pm\") pod \"e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.688469 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.688433 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:15.877928 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:15.877827 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:16.007792 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:16.007764 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd"] Sep 11 14:35:16.008243 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:35:16.008215 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9740e363_d91f_4efd_b5a6_f487ec62151c.slice/crio-c3a472a5597addbf04dd5f31b1839ebad7331dd319ad6f79b7dd8f26184899cd WatchSource:0}: Error finding container c3a472a5597addbf04dd5f31b1839ebad7331dd319ad6f79b7dd8f26184899cd: Status 404 returned error can't find the container with id c3a472a5597addbf04dd5f31b1839ebad7331dd319ad6f79b7dd8f26184899cd Sep 11 14:35:16.010666 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:16.010632 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:35:16.010748 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:16.010700 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:35:16.010748 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:16.010732 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:35:16.292986 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:16.292946 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" event={"ID":"9740e363-d91f-4efd-b5a6-f487ec62151c","Type":"ContainerStarted","Data":"da24dc9d4bff35519f0370ab4797d933d52e1bf39c6505772ea53ade05f7ca39"} Sep 11 14:35:16.293150 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:16.292998 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" event={"ID":"9740e363-d91f-4efd-b5a6-f487ec62151c","Type":"ContainerStarted","Data":"c3a472a5597addbf04dd5f31b1839ebad7331dd319ad6f79b7dd8f26184899cd"} Sep 11 14:35:16.688524 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:16.688485 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:17.360129 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:17.360080 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:17.693939 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:17.693908 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:17.730088 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:17.730028 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" podStartSLOduration=2.73000864 podStartE2EDuration="2.73000864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:35:16.359765797 +0000 UTC m=+1226.973872143" watchObservedRunningTime="2026-09-11 14:35:17.73000864 +0000 UTC m=+1228.344114984" Sep 11 14:35:18.303576 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:18.303546 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:18.304675 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:18.304655 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:18.541091 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:18.541056 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:18.660757 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:18.660722 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:18.858544 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:18.858404 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:20.439958 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:20.439923 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:20.593540 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:20.593488 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:21.123028 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:21.122989 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:22.296389 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:22.294055 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:22.702581 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:22.702546 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:23.333059 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:23.333026 2553 generic.go:356] "Generic (PLEG): container finished" podID="e5e92780-d597-45c4-824d-9b453e24d070" containerID="55b5aa39b15b2800054928f630bf2f4ac0079701ed3187ca21f47858cca489c8" exitCode=0 Sep 11 14:35:23.333458 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:23.333094 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" event={"ID":"e5e92780-d597-45c4-824d-9b453e24d070","Type":"ContainerDied","Data":"55b5aa39b15b2800054928f630bf2f4ac0079701ed3187ca21f47858cca489c8"} Sep 11 14:35:23.333539 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:23.333519 2553 scope.go:117] "RemoveContainer" containerID="55b5aa39b15b2800054928f630bf2f4ac0079701ed3187ca21f47858cca489c8" Sep 11 14:35:23.334618 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:23.334592 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:35:23.334686 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:23.334656 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:35:23.334722 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:23.334684 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:35:23.545958 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:23.545915 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs"] Sep 11 14:35:23.549805 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:23.549778 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs" Sep 11 14:35:23.553756 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:23.553728 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad70543-ac3a-4db8-bf3b-d31c774235fe-plugins-config-volume\") pod \"payload-processing-e2e-derive-15594dcc-687885c585-2vrfs\" (UID: \"1ad70543-ac3a-4db8-bf3b-d31c774235fe\") " pod="openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs" Sep 11 14:35:23.553878 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:23.553810 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqfwx\" (UniqueName: \"kubernetes.io/projected/1ad70543-ac3a-4db8-bf3b-d31c774235fe-kube-api-access-dqfwx\") pod \"payload-processing-e2e-derive-15594dcc-687885c585-2vrfs\" (UID: \"1ad70543-ac3a-4db8-bf3b-d31c774235fe\") " pod="openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs" Sep 11 14:35:23.560837 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:23.560813 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-15594dcc-dockercfg-6vr2v\"" Sep 11 14:35:23.560963 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:23.560948 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-15594dcc\"" Sep 11 14:35:23.578568 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:23.578544 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs"] Sep 11 14:35:23.612791 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:23.612706 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:23.655143 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:23.655100 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad70543-ac3a-4db8-bf3b-d31c774235fe-plugins-config-volume\") pod \"payload-processing-e2e-derive-15594dcc-687885c585-2vrfs\" (UID: \"1ad70543-ac3a-4db8-bf3b-d31c774235fe\") " pod="openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs" Sep 11 14:35:23.655318 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:23.655182 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqfwx\" (UniqueName: \"kubernetes.io/projected/1ad70543-ac3a-4db8-bf3b-d31c774235fe-kube-api-access-dqfwx\") pod \"payload-processing-e2e-derive-15594dcc-687885c585-2vrfs\" (UID: \"1ad70543-ac3a-4db8-bf3b-d31c774235fe\") " pod="openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs" Sep 11 14:35:23.656079 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:23.656056 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad70543-ac3a-4db8-bf3b-d31c774235fe-plugins-config-volume\") pod \"payload-processing-e2e-derive-15594dcc-687885c585-2vrfs\" (UID: \"1ad70543-ac3a-4db8-bf3b-d31c774235fe\") " pod="openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs" Sep 11 14:35:23.669002 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:23.668971 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqfwx\" (UniqueName: \"kubernetes.io/projected/1ad70543-ac3a-4db8-bf3b-d31c774235fe-kube-api-access-dqfwx\") pod \"payload-processing-e2e-derive-15594dcc-687885c585-2vrfs\" (UID: \"1ad70543-ac3a-4db8-bf3b-d31c774235fe\") " pod="openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs" Sep 11 14:35:23.862731 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:23.862692 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs" Sep 11 14:35:24.181910 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:24.181886 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs"] Sep 11 14:35:24.182883 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:35:24.182858 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad70543_ac3a_4db8_bf3b_d31c774235fe.slice/crio-2bd9dd90129d131866ce1096a9a28922e621dd22cfa78813660f674c2c7931f1 WatchSource:0}: Error finding container 2bd9dd90129d131866ce1096a9a28922e621dd22cfa78813660f674c2c7931f1: Status 404 returned error can't find the container with id 2bd9dd90129d131866ce1096a9a28922e621dd22cfa78813660f674c2c7931f1 Sep 11 14:35:24.314226 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:24.314193 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:24.340497 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:24.340460 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs" event={"ID":"1ad70543-ac3a-4db8-bf3b-d31c774235fe","Type":"ContainerStarted","Data":"e3bf452e83c91b35f567b17bdf886d62ef8615fadf9d541d4b9bf8204802eb92"} Sep 11 14:35:24.341046 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:24.340524 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs" event={"ID":"1ad70543-ac3a-4db8-bf3b-d31c774235fe","Type":"ContainerStarted","Data":"2bd9dd90129d131866ce1096a9a28922e621dd22cfa78813660f674c2c7931f1"} Sep 11 14:35:24.341046 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:24.340638 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs" Sep 11 14:35:24.342775 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:24.342748 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" event={"ID":"e5e92780-d597-45c4-824d-9b453e24d070","Type":"ContainerStarted","Data":"63aa596f645b3cdf083b3a4160d0ccfdbc828584e3768e633ecf734204196661"} Sep 11 14:35:24.409800 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:24.409743 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs" podStartSLOduration=1.409728031 podStartE2EDuration="1.409728031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:35:24.408664631 +0000 UTC m=+1235.022770974" watchObservedRunningTime="2026-09-11 14:35:24.409728031 +0000 UTC m=+1235.023834375" Sep 11 14:35:24.421808 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:24.421775 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:24.422062 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:24.422036 2553 patch_prober.go:28] interesting pod/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 11 14:35:24.422124 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:24.422086 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" podUID="e5e92780-d597-45c4-824d-9b453e24d070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 11 14:35:25.421809 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:25.421774 2553 patch_prober.go:28] interesting pod/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 11 14:35:25.422185 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:25.421829 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" podUID="e5e92780-d597-45c4-824d-9b453e24d070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 11 14:35:25.755248 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:25.755152 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:26.422659 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:26.422625 2553 patch_prober.go:28] interesting pod/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 11 14:35:26.423027 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:26.422692 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" podUID="e5e92780-d597-45c4-824d-9b453e24d070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 11 14:35:27.422053 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:27.422014 2553 patch_prober.go:28] interesting pod/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 11 14:35:27.422239 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:27.422083 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" podUID="e5e92780-d597-45c4-824d-9b453e24d070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 11 14:35:27.994299 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:27.994261 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:28.422470 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:28.422440 2553 patch_prober.go:28] interesting pod/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 11 14:35:28.422641 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:28.422491 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" podUID="e5e92780-d597-45c4-824d-9b453e24d070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 11 14:35:29.422471 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:29.422407 2553 patch_prober.go:28] interesting pod/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Sep 11 14:35:29.422850 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:29.422503 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" podUID="e5e92780-d597-45c4-824d-9b453e24d070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Sep 11 14:35:29.709459 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:29.709341 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:30.479878 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:30.479842 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:30.480393 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:30.480331 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:30.481982 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:30.481959 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:35:31.191562 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:31.191528 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:33.501960 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:33.501923 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:34.763801 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:34.763745 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:35.353100 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:35.353071 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs" Sep 11 14:35:36.487588 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:36.487555 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:38.282425 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:38.282381 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:39.996277 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:39.996241 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:43.762952 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:43.762910 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:43.834863 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:43.834829 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs"] Sep 11 14:35:43.835103 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:43.835077 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs" podUID="1ad70543-ac3a-4db8-bf3b-d31c774235fe" containerName="payload-processing" containerID="cri-o://e3bf452e83c91b35f567b17bdf886d62ef8615fadf9d541d4b9bf8204802eb92" gracePeriod=30 Sep 11 14:35:44.097547 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:44.097520 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs" Sep 11 14:35:44.152385 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:44.152348 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad70543-ac3a-4db8-bf3b-d31c774235fe-plugins-config-volume\") pod \"1ad70543-ac3a-4db8-bf3b-d31c774235fe\" (UID: \"1ad70543-ac3a-4db8-bf3b-d31c774235fe\") " Sep 11 14:35:44.152589 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:44.152395 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqfwx\" (UniqueName: \"kubernetes.io/projected/1ad70543-ac3a-4db8-bf3b-d31c774235fe-kube-api-access-dqfwx\") pod \"1ad70543-ac3a-4db8-bf3b-d31c774235fe\" (UID: \"1ad70543-ac3a-4db8-bf3b-d31c774235fe\") " Sep 11 14:35:44.152806 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:44.152780 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ad70543-ac3a-4db8-bf3b-d31c774235fe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ad70543-ac3a-4db8-bf3b-d31c774235fe" (UID: "1ad70543-ac3a-4db8-bf3b-d31c774235fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:35:44.152960 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:44.152944 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad70543-ac3a-4db8-bf3b-d31c774235fe-plugins-config-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:35:44.154652 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:44.154620 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad70543-ac3a-4db8-bf3b-d31c774235fe-kube-api-access-dqfwx" (OuterVolumeSpecName: "kube-api-access-dqfwx") pod "1ad70543-ac3a-4db8-bf3b-d31c774235fe" (UID: "1ad70543-ac3a-4db8-bf3b-d31c774235fe"). InnerVolumeSpecName "kube-api-access-dqfwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:35:44.254389 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:44.254347 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqfwx\" (UniqueName: \"kubernetes.io/projected/1ad70543-ac3a-4db8-bf3b-d31c774235fe-kube-api-access-dqfwx\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:35:44.457753 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:44.457719 2553 generic.go:356] "Generic (PLEG): container finished" podID="1ad70543-ac3a-4db8-bf3b-d31c774235fe" containerID="e3bf452e83c91b35f567b17bdf886d62ef8615fadf9d541d4b9bf8204802eb92" exitCode=0 Sep 11 14:35:44.457917 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:44.457769 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs" Sep 11 14:35:44.457917 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:44.457776 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs" event={"ID":"1ad70543-ac3a-4db8-bf3b-d31c774235fe","Type":"ContainerDied","Data":"e3bf452e83c91b35f567b17bdf886d62ef8615fadf9d541d4b9bf8204802eb92"} Sep 11 14:35:44.457917 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:44.457829 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs" event={"ID":"1ad70543-ac3a-4db8-bf3b-d31c774235fe","Type":"ContainerDied","Data":"2bd9dd90129d131866ce1096a9a28922e621dd22cfa78813660f674c2c7931f1"} Sep 11 14:35:44.457917 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:44.457860 2553 scope.go:117] "RemoveContainer" containerID="e3bf452e83c91b35f567b17bdf886d62ef8615fadf9d541d4b9bf8204802eb92" Sep 11 14:35:44.467501 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:44.467470 2553 scope.go:117] "RemoveContainer" containerID="e3bf452e83c91b35f567b17bdf886d62ef8615fadf9d541d4b9bf8204802eb92" Sep 11 14:35:44.467795 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:35:44.467767 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3bf452e83c91b35f567b17bdf886d62ef8615fadf9d541d4b9bf8204802eb92\": container with ID starting with e3bf452e83c91b35f567b17bdf886d62ef8615fadf9d541d4b9bf8204802eb92 not found: ID does not exist" containerID="e3bf452e83c91b35f567b17bdf886d62ef8615fadf9d541d4b9bf8204802eb92" Sep 11 14:35:44.467865 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:44.467809 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3bf452e83c91b35f567b17bdf886d62ef8615fadf9d541d4b9bf8204802eb92"} err="failed to get container status \"e3bf452e83c91b35f567b17bdf886d62ef8615fadf9d541d4b9bf8204802eb92\": rpc error: code = NotFound desc = could not find container \"e3bf452e83c91b35f567b17bdf886d62ef8615fadf9d541d4b9bf8204802eb92\": container with ID starting with e3bf452e83c91b35f567b17bdf886d62ef8615fadf9d541d4b9bf8204802eb92 not found: ID does not exist" Sep 11 14:35:44.499374 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:44.499319 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs"] Sep 11 14:35:44.535723 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:44.535693 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15594dcc-687885c585-2vrfs"] Sep 11 14:35:44.571088 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:44.571046 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd"] Sep 11 14:35:44.571353 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:44.571324 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" podUID="9740e363-d91f-4efd-b5a6-f487ec62151c" containerName="istio-proxy" containerID="cri-o://da24dc9d4bff35519f0370ab4797d933d52e1bf39c6505772ea53ade05f7ca39" gracePeriod=30 Sep 11 14:35:45.184143 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.184071 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:45.222053 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.222031 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:45.263911 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.263835 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9740e363-d91f-4efd-b5a6-f487ec62151c-istiod-ca-cert\") pod \"9740e363-d91f-4efd-b5a6-f487ec62151c\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " Sep 11 14:35:45.263911 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.263866 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-envoy\") pod \"9740e363-d91f-4efd-b5a6-f487ec62151c\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " Sep 11 14:35:45.264127 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.263914 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-podinfo\") pod \"9740e363-d91f-4efd-b5a6-f487ec62151c\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " Sep 11 14:35:45.264127 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.263937 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-credential-socket\") pod \"9740e363-d91f-4efd-b5a6-f487ec62151c\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " Sep 11 14:35:45.264127 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.264012 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p24pm\" (UniqueName: \"kubernetes.io/projected/9740e363-d91f-4efd-b5a6-f487ec62151c-kube-api-access-p24pm\") pod \"9740e363-d91f-4efd-b5a6-f487ec62151c\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " Sep 11 14:35:45.264127 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.264077 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-data\") pod \"9740e363-d91f-4efd-b5a6-f487ec62151c\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " Sep 11 14:35:45.264127 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.264110 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-workload-socket\") pod \"9740e363-d91f-4efd-b5a6-f487ec62151c\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " Sep 11 14:35:45.264386 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.264153 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-workload-certs\") pod \"9740e363-d91f-4efd-b5a6-f487ec62151c\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " Sep 11 14:35:45.264386 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.264178 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-token\") pod \"9740e363-d91f-4efd-b5a6-f487ec62151c\" (UID: \"9740e363-d91f-4efd-b5a6-f487ec62151c\") " Sep 11 14:35:45.264386 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.264272 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9740e363-d91f-4efd-b5a6-f487ec62151c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9740e363-d91f-4efd-b5a6-f487ec62151c" (UID: "9740e363-d91f-4efd-b5a6-f487ec62151c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:35:45.264386 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.264272 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9740e363-d91f-4efd-b5a6-f487ec62151c" (UID: "9740e363-d91f-4efd-b5a6-f487ec62151c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:35:45.264627 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.264478 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9740e363-d91f-4efd-b5a6-f487ec62151c" (UID: "9740e363-d91f-4efd-b5a6-f487ec62151c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:35:45.264627 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.264503 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9740e363-d91f-4efd-b5a6-f487ec62151c" (UID: "9740e363-d91f-4efd-b5a6-f487ec62151c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:35:45.264627 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.264586 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-workload-socket\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:35:45.264627 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.264601 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-workload-certs\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:35:45.264838 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.264617 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9740e363-d91f-4efd-b5a6-f487ec62151c-istiod-ca-cert\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:35:45.264838 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.264653 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-credential-socket\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:35:45.264838 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.264689 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-data" (OuterVolumeSpecName: "istio-data") pod "9740e363-d91f-4efd-b5a6-f487ec62151c" (UID: "9740e363-d91f-4efd-b5a6-f487ec62151c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:35:45.266225 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.266194 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9740e363-d91f-4efd-b5a6-f487ec62151c" (UID: "9740e363-d91f-4efd-b5a6-f487ec62151c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:35:45.266225 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.266198 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9740e363-d91f-4efd-b5a6-f487ec62151c" (UID: "9740e363-d91f-4efd-b5a6-f487ec62151c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:35:45.266391 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.266263 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9740e363-d91f-4efd-b5a6-f487ec62151c-kube-api-access-p24pm" (OuterVolumeSpecName: "kube-api-access-p24pm") pod "9740e363-d91f-4efd-b5a6-f487ec62151c" (UID: "9740e363-d91f-4efd-b5a6-f487ec62151c"). InnerVolumeSpecName "kube-api-access-p24pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:35:45.266391 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.266378 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-token" (OuterVolumeSpecName: "istio-token") pod "9740e363-d91f-4efd-b5a6-f487ec62151c" (UID: "9740e363-d91f-4efd-b5a6-f487ec62151c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:35:45.305638 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.305600 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:45.365579 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.365545 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-data\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:35:45.365579 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.365571 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-token\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:35:45.365579 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.365579 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-envoy\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:35:45.365579 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.365588 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9740e363-d91f-4efd-b5a6-f487ec62151c-istio-podinfo\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:35:45.365842 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.365598 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p24pm\" (UniqueName: \"kubernetes.io/projected/9740e363-d91f-4efd-b5a6-f487ec62151c-kube-api-access-p24pm\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:35:45.464821 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.464783 2553 generic.go:356] "Generic (PLEG): container finished" podID="9740e363-d91f-4efd-b5a6-f487ec62151c" containerID="da24dc9d4bff35519f0370ab4797d933d52e1bf39c6505772ea53ade05f7ca39" exitCode=0 Sep 11 14:35:45.464996 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.464839 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" Sep 11 14:35:45.464996 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.464848 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" event={"ID":"9740e363-d91f-4efd-b5a6-f487ec62151c","Type":"ContainerDied","Data":"da24dc9d4bff35519f0370ab4797d933d52e1bf39c6505772ea53ade05f7ca39"} Sep 11 14:35:45.464996 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.464901 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd" event={"ID":"9740e363-d91f-4efd-b5a6-f487ec62151c","Type":"ContainerDied","Data":"c3a472a5597addbf04dd5f31b1839ebad7331dd319ad6f79b7dd8f26184899cd"} Sep 11 14:35:45.464996 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.464929 2553 scope.go:117] "RemoveContainer" containerID="da24dc9d4bff35519f0370ab4797d933d52e1bf39c6505772ea53ade05f7ca39" Sep 11 14:35:45.473843 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.473825 2553 scope.go:117] "RemoveContainer" containerID="da24dc9d4bff35519f0370ab4797d933d52e1bf39c6505772ea53ade05f7ca39" Sep 11 14:35:45.474118 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:35:45.474099 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da24dc9d4bff35519f0370ab4797d933d52e1bf39c6505772ea53ade05f7ca39\": container with ID starting with da24dc9d4bff35519f0370ab4797d933d52e1bf39c6505772ea53ade05f7ca39 not found: ID does not exist" containerID="da24dc9d4bff35519f0370ab4797d933d52e1bf39c6505772ea53ade05f7ca39" Sep 11 14:35:45.474208 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.474125 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da24dc9d4bff35519f0370ab4797d933d52e1bf39c6505772ea53ade05f7ca39"} err="failed to get container status \"da24dc9d4bff35519f0370ab4797d933d52e1bf39c6505772ea53ade05f7ca39\": rpc error: code = NotFound desc = could not find container \"da24dc9d4bff35519f0370ab4797d933d52e1bf39c6505772ea53ade05f7ca39\": container with ID starting with da24dc9d4bff35519f0370ab4797d933d52e1bf39c6505772ea53ade05f7ca39 not found: ID does not exist" Sep 11 14:35:45.480452 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.480429 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:45.497931 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.497907 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd"] Sep 11 14:35:45.510245 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.510222 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-15594dcc-openshift-default-5f86c9748d-98rgd"] Sep 11 14:35:45.623230 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.623135 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:45.912329 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.912294 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ad70543-ac3a-4db8-bf3b-d31c774235fe" path="/var/lib/kubelet/pods/1ad70543-ac3a-4db8-bf3b-d31c774235fe/volumes" Sep 11 14:35:45.912741 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:45.912727 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9740e363-d91f-4efd-b5a6-f487ec62151c" path="/var/lib/kubelet/pods/9740e363-d91f-4efd-b5a6-f487ec62151c/volumes" Sep 11 14:35:47.279619 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:47.279586 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:47.461758 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:47.461724 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:48.886863 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:48.886833 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:50.118850 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:50.118810 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:51.208821 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.208786 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q"] Sep 11 14:35:51.209231 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.209180 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9740e363-d91f-4efd-b5a6-f487ec62151c" containerName="istio-proxy" Sep 11 14:35:51.209231 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.209191 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9740e363-d91f-4efd-b5a6-f487ec62151c" containerName="istio-proxy" Sep 11 14:35:51.209231 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.209198 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ad70543-ac3a-4db8-bf3b-d31c774235fe" containerName="payload-processing" Sep 11 14:35:51.209231 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.209204 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ad70543-ac3a-4db8-bf3b-d31c774235fe" containerName="payload-processing" Sep 11 14:35:51.209384 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.209271 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9740e363-d91f-4efd-b5a6-f487ec62151c" containerName="istio-proxy" Sep 11 14:35:51.209384 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.209281 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ad70543-ac3a-4db8-bf3b-d31c774235fe" containerName="payload-processing" Sep 11 14:35:51.213705 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.213683 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:35:51.236698 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.236676 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q"] Sep 11 14:35:51.294564 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.294537 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5b0917\"/\"test-model-5b0917-kserve-self-signed-certs\"" Sep 11 14:35:51.294979 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.294960 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5b0917\"/\"kube-root-ca.crt\"" Sep 11 14:35:51.295047 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.295002 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5b0917\"/\"default-dockercfg-jcbfd\"" Sep 11 14:35:51.295219 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.295193 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5b0917\"/\"openshift-service-ca.crt\"" Sep 11 14:35:51.306572 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.306547 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:51.320358 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.320333 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfj8v\" (UniqueName: \"kubernetes.io/projected/153b89ca-92c3-4562-b984-97c8af35ac2b-kube-api-access-pfj8v\") pod \"test-model-5b0917-kserve-7865c64589-kwc2q\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:35:51.320483 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.320384 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-model-cache\") pod \"test-model-5b0917-kserve-7865c64589-kwc2q\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:35:51.320532 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.320479 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-tmp-dir\") pod \"test-model-5b0917-kserve-7865c64589-kwc2q\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:35:51.320568 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.320554 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-dshm\") pod \"test-model-5b0917-kserve-7865c64589-kwc2q\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:35:51.320633 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.320615 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/153b89ca-92c3-4562-b984-97c8af35ac2b-tls-certs\") pod \"test-model-5b0917-kserve-7865c64589-kwc2q\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:35:51.320673 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.320650 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-home\") pod \"test-model-5b0917-kserve-7865c64589-kwc2q\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:35:51.421389 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.421359 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-tmp-dir\") pod \"test-model-5b0917-kserve-7865c64589-kwc2q\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:35:51.421619 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.421476 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-dshm\") pod \"test-model-5b0917-kserve-7865c64589-kwc2q\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:35:51.421619 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.421520 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/153b89ca-92c3-4562-b984-97c8af35ac2b-tls-certs\") pod \"test-model-5b0917-kserve-7865c64589-kwc2q\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:35:51.421619 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.421549 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-home\") pod \"test-model-5b0917-kserve-7865c64589-kwc2q\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:35:51.421619 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.421582 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfj8v\" (UniqueName: \"kubernetes.io/projected/153b89ca-92c3-4562-b984-97c8af35ac2b-kube-api-access-pfj8v\") pod \"test-model-5b0917-kserve-7865c64589-kwc2q\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:35:51.421841 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.421631 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-model-cache\") pod \"test-model-5b0917-kserve-7865c64589-kwc2q\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:35:51.421900 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.421841 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-tmp-dir\") pod \"test-model-5b0917-kserve-7865c64589-kwc2q\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:35:51.422043 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.421897 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-home\") pod \"test-model-5b0917-kserve-7865c64589-kwc2q\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:35:51.422043 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.421998 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-model-cache\") pod \"test-model-5b0917-kserve-7865c64589-kwc2q\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:35:51.423629 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.423609 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-dshm\") pod \"test-model-5b0917-kserve-7865c64589-kwc2q\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:35:51.423886 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.423869 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/153b89ca-92c3-4562-b984-97c8af35ac2b-tls-certs\") pod \"test-model-5b0917-kserve-7865c64589-kwc2q\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:35:51.445511 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.445485 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfj8v\" (UniqueName: \"kubernetes.io/projected/153b89ca-92c3-4562-b984-97c8af35ac2b-kube-api-access-pfj8v\") pod \"test-model-5b0917-kserve-7865c64589-kwc2q\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:35:51.523126 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.523036 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:35:51.818840 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:51.818817 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q"] Sep 11 14:35:51.821027 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:35:51.821002 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod153b89ca_92c3_4562_b984_97c8af35ac2b.slice/crio-27a0e37b803e90bde4400d9f7222fde0505e5647a7ec1fc7402c89232ee7955f WatchSource:0}: Error finding container 27a0e37b803e90bde4400d9f7222fde0505e5647a7ec1fc7402c89232ee7955f: Status 404 returned error can't find the container with id 27a0e37b803e90bde4400d9f7222fde0505e5647a7ec1fc7402c89232ee7955f Sep 11 14:35:52.510583 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:52.510549 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" event={"ID":"153b89ca-92c3-4562-b984-97c8af35ac2b","Type":"ContainerStarted","Data":"57e4214f6d23a3c0dceac038d195977b9b2695d7fdf9a8f852384528e999ec99"} Sep 11 14:35:52.510583 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:52.510588 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" event={"ID":"153b89ca-92c3-4562-b984-97c8af35ac2b","Type":"ContainerStarted","Data":"27a0e37b803e90bde4400d9f7222fde0505e5647a7ec1fc7402c89232ee7955f"} Sep 11 14:35:52.554930 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:52.554880 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" podStartSLOduration=1.554863208 podStartE2EDuration="1.554863208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:35:52.55205998 +0000 UTC m=+1263.166166321" watchObservedRunningTime="2026-09-11 14:35:52.554863208 +0000 UTC m=+1263.168969552" Sep 11 14:35:53.397916 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:53.397881 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:54.491607 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:54.491568 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:54.669148 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:54.669104 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:54.815605 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:54.815500 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:35:55.507380 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:35:55.507347 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:01.523945 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:01.523912 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:36:01.524451 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:01.523960 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:36:01.536492 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:01.536470 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:36:01.569398 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:01.569376 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:36:02.660911 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:02.660869 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:12.558164 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:12.558117 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:18.957552 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:18.957517 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr"] Sep 11 14:36:18.960978 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:18.960960 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:18.972034 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:18.972003 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-69a7cf\"/\"kube-root-ca.crt\"" Sep 11 14:36:18.973523 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:18.973504 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-69a7cf\"/\"default-dockercfg-hc52x\"" Sep 11 14:36:18.973619 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:18.973527 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-69a7cf\"/\"explicit-model-69a7cf-kserve-self-signed-certs\"" Sep 11 14:36:18.973619 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:18.973504 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-69a7cf\"/\"openshift-service-ca.crt\"" Sep 11 14:36:18.974988 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:18.974971 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-home\") pod \"explicit-model-69a7cf-kserve-9575c4fd5-d65jr\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:18.975061 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:18.974995 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-dshm\") pod \"explicit-model-69a7cf-kserve-9575c4fd5-d65jr\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:18.975061 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:18.975019 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfhqs\" (UniqueName: \"kubernetes.io/projected/2b1caf7a-9b03-441d-8d50-70805c09b911-kube-api-access-xfhqs\") pod \"explicit-model-69a7cf-kserve-9575c4fd5-d65jr\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:18.975061 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:18.975036 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b1caf7a-9b03-441d-8d50-70805c09b911-tls-certs\") pod \"explicit-model-69a7cf-kserve-9575c4fd5-d65jr\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:18.975166 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:18.975141 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-model-cache\") pod \"explicit-model-69a7cf-kserve-9575c4fd5-d65jr\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:18.975219 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:18.975204 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-tmp-dir\") pod \"explicit-model-69a7cf-kserve-9575c4fd5-d65jr\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:18.999582 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:18.999550 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr"] Sep 11 14:36:19.018273 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:19.018242 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:19.075732 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:19.075691 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-tmp-dir\") pod \"explicit-model-69a7cf-kserve-9575c4fd5-d65jr\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:19.075898 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:19.075828 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-home\") pod \"explicit-model-69a7cf-kserve-9575c4fd5-d65jr\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:19.075898 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:19.075846 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-dshm\") pod \"explicit-model-69a7cf-kserve-9575c4fd5-d65jr\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:19.075898 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:19.075871 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfhqs\" (UniqueName: \"kubernetes.io/projected/2b1caf7a-9b03-441d-8d50-70805c09b911-kube-api-access-xfhqs\") pod \"explicit-model-69a7cf-kserve-9575c4fd5-d65jr\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:19.075898 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:19.075888 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b1caf7a-9b03-441d-8d50-70805c09b911-tls-certs\") pod \"explicit-model-69a7cf-kserve-9575c4fd5-d65jr\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:19.076093 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:19.076051 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-model-cache\") pod \"explicit-model-69a7cf-kserve-9575c4fd5-d65jr\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:19.076164 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:19.076143 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-tmp-dir\") pod \"explicit-model-69a7cf-kserve-9575c4fd5-d65jr\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:19.076241 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:19.076221 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-home\") pod \"explicit-model-69a7cf-kserve-9575c4fd5-d65jr\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:19.076383 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:19.076365 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-model-cache\") pod \"explicit-model-69a7cf-kserve-9575c4fd5-d65jr\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:19.078094 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:19.078077 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-dshm\") pod \"explicit-model-69a7cf-kserve-9575c4fd5-d65jr\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:19.078383 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:19.078365 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b1caf7a-9b03-441d-8d50-70805c09b911-tls-certs\") pod \"explicit-model-69a7cf-kserve-9575c4fd5-d65jr\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:19.092688 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:19.092658 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfhqs\" (UniqueName: \"kubernetes.io/projected/2b1caf7a-9b03-441d-8d50-70805c09b911-kube-api-access-xfhqs\") pod \"explicit-model-69a7cf-kserve-9575c4fd5-d65jr\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:19.270922 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:19.270830 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:19.561335 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:19.561308 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr"] Sep 11 14:36:19.646155 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:19.646121 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" event={"ID":"2b1caf7a-9b03-441d-8d50-70805c09b911","Type":"ContainerStarted","Data":"a9feb67961dfe33ff38a3f70f020c0d59f01fc85321f3a36cc5cf68f4b2b92dc"} Sep 11 14:36:19.646338 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:19.646165 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" event={"ID":"2b1caf7a-9b03-441d-8d50-70805c09b911","Type":"ContainerStarted","Data":"0ff6e79e0ffd88fd4a004498b355861f267e56f2c072a2e7e55ae8687a046eff"} Sep 11 14:36:19.673066 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:19.673007 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" podStartSLOduration=1.672990107 podStartE2EDuration="1.672990107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:36:19.670737416 +0000 UTC m=+1290.284843769" watchObservedRunningTime="2026-09-11 14:36:19.672990107 +0000 UTC m=+1290.287096451" Sep 11 14:36:21.441228 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:21.441190 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:29.144453 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:29.144401 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:29.271762 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:29.271727 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:29.271955 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:29.271773 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:29.284588 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:29.284559 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:29.285565 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:29.285541 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:29.383917 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:29.383885 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:29.708125 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:29.708093 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:37.551371 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:37.551333 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr"] Sep 11 14:36:37.585533 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:37.585502 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:37.876986 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:37.876889 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:39.136957 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:39.136925 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:40.352326 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:40.352294 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:40.799687 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:40.799652 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:41.322513 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:41.322470 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:42.674217 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:42.674166 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:45.985561 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:45.985524 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:48.573440 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:48.573391 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:48.707592 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:48.707555 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:48.868561 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:48.868465 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:49.004656 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:49.004614 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:49.110345 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:49.110312 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:49.415194 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:49.415156 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:49.521583 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:49.521546 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:50.520538 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:50.520503 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:51.657406 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:51.657360 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:52.570688 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:52.570628 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" podUID="2b1caf7a-9b03-441d-8d50-70805c09b911" containerName="main" containerID="cri-o://a9feb67961dfe33ff38a3f70f020c0d59f01fc85321f3a36cc5cf68f4b2b92dc" gracePeriod=45 Sep 11 14:36:52.813943 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:52.813912 2553 generic.go:356] "Generic (PLEG): container finished" podID="2b1caf7a-9b03-441d-8d50-70805c09b911" containerID="a9feb67961dfe33ff38a3f70f020c0d59f01fc85321f3a36cc5cf68f4b2b92dc" exitCode=0 Sep 11 14:36:52.814283 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:52.813982 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" event={"ID":"2b1caf7a-9b03-441d-8d50-70805c09b911","Type":"ContainerDied","Data":"a9feb67961dfe33ff38a3f70f020c0d59f01fc85321f3a36cc5cf68f4b2b92dc"} Sep 11 14:36:52.814283 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:52.814019 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" event={"ID":"2b1caf7a-9b03-441d-8d50-70805c09b911","Type":"ContainerDied","Data":"0ff6e79e0ffd88fd4a004498b355861f267e56f2c072a2e7e55ae8687a046eff"} Sep 11 14:36:52.814283 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:52.814031 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ff6e79e0ffd88fd4a004498b355861f267e56f2c072a2e7e55ae8687a046eff" Sep 11 14:36:52.821453 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:52.821373 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:53.014736 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:53.014702 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-model-cache\") pod \"2b1caf7a-9b03-441d-8d50-70805c09b911\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " Sep 11 14:36:53.014937 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:53.014762 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-tmp-dir\") pod \"2b1caf7a-9b03-441d-8d50-70805c09b911\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " Sep 11 14:36:53.014937 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:53.014808 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfhqs\" (UniqueName: \"kubernetes.io/projected/2b1caf7a-9b03-441d-8d50-70805c09b911-kube-api-access-xfhqs\") pod \"2b1caf7a-9b03-441d-8d50-70805c09b911\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " Sep 11 14:36:53.014937 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:53.014842 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b1caf7a-9b03-441d-8d50-70805c09b911-tls-certs\") pod \"2b1caf7a-9b03-441d-8d50-70805c09b911\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " Sep 11 14:36:53.014937 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:53.014892 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-home\") pod \"2b1caf7a-9b03-441d-8d50-70805c09b911\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " Sep 11 14:36:53.014937 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:53.014922 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-dshm\") pod \"2b1caf7a-9b03-441d-8d50-70805c09b911\" (UID: \"2b1caf7a-9b03-441d-8d50-70805c09b911\") " Sep 11 14:36:53.015387 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:53.015084 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-model-cache" (OuterVolumeSpecName: "model-cache") pod "2b1caf7a-9b03-441d-8d50-70805c09b911" (UID: "2b1caf7a-9b03-441d-8d50-70805c09b911"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:36:53.015387 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:53.015120 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2b1caf7a-9b03-441d-8d50-70805c09b911" (UID: "2b1caf7a-9b03-441d-8d50-70805c09b911"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:36:53.015387 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:53.015194 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-home" (OuterVolumeSpecName: "home") pod "2b1caf7a-9b03-441d-8d50-70805c09b911" (UID: "2b1caf7a-9b03-441d-8d50-70805c09b911"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:36:53.015387 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:53.015260 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-home\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:36:53.015387 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:53.015278 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-model-cache\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:36:53.015387 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:53.015296 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-tmp-dir\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:36:53.017280 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:53.017252 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-dshm" (OuterVolumeSpecName: "dshm") pod "2b1caf7a-9b03-441d-8d50-70805c09b911" (UID: "2b1caf7a-9b03-441d-8d50-70805c09b911"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:36:53.017386 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:53.017323 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b1caf7a-9b03-441d-8d50-70805c09b911-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b1caf7a-9b03-441d-8d50-70805c09b911" (UID: "2b1caf7a-9b03-441d-8d50-70805c09b911"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:36:53.017386 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:53.017336 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1caf7a-9b03-441d-8d50-70805c09b911-kube-api-access-xfhqs" (OuterVolumeSpecName: "kube-api-access-xfhqs") pod "2b1caf7a-9b03-441d-8d50-70805c09b911" (UID: "2b1caf7a-9b03-441d-8d50-70805c09b911"). InnerVolumeSpecName "kube-api-access-xfhqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:36:53.116386 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:53.116303 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b1caf7a-9b03-441d-8d50-70805c09b911-dshm\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:36:53.116386 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:53.116331 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfhqs\" (UniqueName: \"kubernetes.io/projected/2b1caf7a-9b03-441d-8d50-70805c09b911-kube-api-access-xfhqs\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:36:53.116386 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:53.116341 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b1caf7a-9b03-441d-8d50-70805c09b911-tls-certs\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:36:53.818513 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:53.818484 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr" Sep 11 14:36:53.875047 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:53.874966 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr"] Sep 11 14:36:53.889159 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:53.889128 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-69a7cf/explicit-model-69a7cf-kserve-9575c4fd5-d65jr"] Sep 11 14:36:53.912506 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:53.912475 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b1caf7a-9b03-441d-8d50-70805c09b911" path="/var/lib/kubelet/pods/2b1caf7a-9b03-441d-8d50-70805c09b911/volumes" Sep 11 14:36:54.791763 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:54.791728 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:54.985921 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:54.985888 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:55.179633 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:55.179587 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:58.372472 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:58.372437 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:58.477603 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:58.477574 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j"] Sep 11 14:36:58.477790 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:58.477768 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" podUID="e5e92780-d597-45c4-824d-9b453e24d070" containerName="istio-proxy" containerID="cri-o://63aa596f645b3cdf083b3a4160d0ccfdbc828584e3768e633ecf734204196661" gracePeriod=30 Sep 11 14:36:58.509248 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:58.509198 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:58.695461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:58.695403 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:59.052854 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:59.052767 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:59.576668 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:59.576632 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:36:59.732227 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:36:59.732191 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:00.482736 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:00.482699 2553 patch_prober.go:28] interesting pod/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:37:00.482910 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:00.482768 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" podUID="e5e92780-d597-45c4-824d-9b453e24d070" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:37:01.437655 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:01.437616 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:01.561178 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:01.561137 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:01.726529 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:01.726437 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:03.724002 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.723976 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:37:03.826163 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.826081 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e92780-d597-45c4-824d-9b453e24d070-istiod-ca-cert\") pod \"e5e92780-d597-45c4-824d-9b453e24d070\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " Sep 11 14:37:03.826163 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.826121 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p88c7\" (UniqueName: \"kubernetes.io/projected/e5e92780-d597-45c4-824d-9b453e24d070-kube-api-access-p88c7\") pod \"e5e92780-d597-45c4-824d-9b453e24d070\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " Sep 11 14:37:03.826163 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.826157 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-credential-socket\") pod \"e5e92780-d597-45c4-824d-9b453e24d070\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " Sep 11 14:37:03.826462 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.826173 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e92780-d597-45c4-824d-9b453e24d070-istio-token\") pod \"e5e92780-d597-45c4-824d-9b453e24d070\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " Sep 11 14:37:03.826462 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.826326 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-istio-envoy\") pod \"e5e92780-d597-45c4-824d-9b453e24d070\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " Sep 11 14:37:03.826462 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.826365 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-workload-certs\") pod \"e5e92780-d597-45c4-824d-9b453e24d070\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " Sep 11 14:37:03.826462 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.826432 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e92780-d597-45c4-824d-9b453e24d070-istio-podinfo\") pod \"e5e92780-d597-45c4-824d-9b453e24d070\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " Sep 11 14:37:03.826686 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.826524 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-workload-socket\") pod \"e5e92780-d597-45c4-824d-9b453e24d070\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " Sep 11 14:37:03.826686 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.826527 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e5e92780-d597-45c4-824d-9b453e24d070" (UID: "e5e92780-d597-45c4-824d-9b453e24d070"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:37:03.826686 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.826535 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e92780-d597-45c4-824d-9b453e24d070-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e5e92780-d597-45c4-824d-9b453e24d070" (UID: "e5e92780-d597-45c4-824d-9b453e24d070"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:37:03.826686 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.826568 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-istio-data\") pod \"e5e92780-d597-45c4-824d-9b453e24d070\" (UID: \"e5e92780-d597-45c4-824d-9b453e24d070\") " Sep 11 14:37:03.826893 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.826804 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e5e92780-d597-45c4-824d-9b453e24d070" (UID: "e5e92780-d597-45c4-824d-9b453e24d070"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:37:03.826959 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.826900 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-credential-socket\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:37:03.826959 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.826888 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-istio-data" (OuterVolumeSpecName: "istio-data") pod "e5e92780-d597-45c4-824d-9b453e24d070" (UID: "e5e92780-d597-45c4-824d-9b453e24d070"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:37:03.826959 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.826920 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-workload-socket\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:37:03.826959 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.826934 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e92780-d597-45c4-824d-9b453e24d070-istiod-ca-cert\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:37:03.826959 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.826934 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e5e92780-d597-45c4-824d-9b453e24d070" (UID: "e5e92780-d597-45c4-824d-9b453e24d070"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:37:03.828451 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.828427 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e92780-d597-45c4-824d-9b453e24d070-kube-api-access-p88c7" (OuterVolumeSpecName: "kube-api-access-p88c7") pod "e5e92780-d597-45c4-824d-9b453e24d070" (UID: "e5e92780-d597-45c4-824d-9b453e24d070"). InnerVolumeSpecName "kube-api-access-p88c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:37:03.828597 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.828552 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e92780-d597-45c4-824d-9b453e24d070-istio-token" (OuterVolumeSpecName: "istio-token") pod "e5e92780-d597-45c4-824d-9b453e24d070" (UID: "e5e92780-d597-45c4-824d-9b453e24d070"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:37:03.828849 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.828834 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e5e92780-d597-45c4-824d-9b453e24d070" (UID: "e5e92780-d597-45c4-824d-9b453e24d070"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:37:03.828923 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.828840 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5e92780-d597-45c4-824d-9b453e24d070-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e5e92780-d597-45c4-824d-9b453e24d070" (UID: "e5e92780-d597-45c4-824d-9b453e24d070"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:37:03.867805 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.867772 2553 generic.go:356] "Generic (PLEG): container finished" podID="e5e92780-d597-45c4-824d-9b453e24d070" containerID="63aa596f645b3cdf083b3a4160d0ccfdbc828584e3768e633ecf734204196661" exitCode=0 Sep 11 14:37:03.867940 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.867826 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" Sep 11 14:37:03.867940 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.867854 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" event={"ID":"e5e92780-d597-45c4-824d-9b453e24d070","Type":"ContainerDied","Data":"63aa596f645b3cdf083b3a4160d0ccfdbc828584e3768e633ecf734204196661"} Sep 11 14:37:03.867940 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.867910 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j" event={"ID":"e5e92780-d597-45c4-824d-9b453e24d070","Type":"ContainerDied","Data":"1f99dff15a154528caa11afc967b21c4049bb6aa10d65a485b819e91e3c65657"} Sep 11 14:37:03.867940 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.867933 2553 scope.go:117] "RemoveContainer" containerID="63aa596f645b3cdf083b3a4160d0ccfdbc828584e3768e633ecf734204196661" Sep 11 14:37:03.878172 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.878154 2553 scope.go:117] "RemoveContainer" containerID="55b5aa39b15b2800054928f630bf2f4ac0079701ed3187ca21f47858cca489c8" Sep 11 14:37:03.885898 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.885881 2553 scope.go:117] "RemoveContainer" containerID="63aa596f645b3cdf083b3a4160d0ccfdbc828584e3768e633ecf734204196661" Sep 11 14:37:03.886161 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:37:03.886144 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63aa596f645b3cdf083b3a4160d0ccfdbc828584e3768e633ecf734204196661\": container with ID starting with 63aa596f645b3cdf083b3a4160d0ccfdbc828584e3768e633ecf734204196661 not found: ID does not exist" containerID="63aa596f645b3cdf083b3a4160d0ccfdbc828584e3768e633ecf734204196661" Sep 11 14:37:03.886212 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.886170 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63aa596f645b3cdf083b3a4160d0ccfdbc828584e3768e633ecf734204196661"} err="failed to get container status \"63aa596f645b3cdf083b3a4160d0ccfdbc828584e3768e633ecf734204196661\": rpc error: code = NotFound desc = could not find container \"63aa596f645b3cdf083b3a4160d0ccfdbc828584e3768e633ecf734204196661\": container with ID starting with 63aa596f645b3cdf083b3a4160d0ccfdbc828584e3768e633ecf734204196661 not found: ID does not exist" Sep 11 14:37:03.886212 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.886189 2553 scope.go:117] "RemoveContainer" containerID="55b5aa39b15b2800054928f630bf2f4ac0079701ed3187ca21f47858cca489c8" Sep 11 14:37:03.886403 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:37:03.886388 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b5aa39b15b2800054928f630bf2f4ac0079701ed3187ca21f47858cca489c8\": container with ID starting with 55b5aa39b15b2800054928f630bf2f4ac0079701ed3187ca21f47858cca489c8 not found: ID does not exist" containerID="55b5aa39b15b2800054928f630bf2f4ac0079701ed3187ca21f47858cca489c8" Sep 11 14:37:03.886469 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.886406 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b5aa39b15b2800054928f630bf2f4ac0079701ed3187ca21f47858cca489c8"} err="failed to get container status \"55b5aa39b15b2800054928f630bf2f4ac0079701ed3187ca21f47858cca489c8\": rpc error: code = NotFound desc = could not find container \"55b5aa39b15b2800054928f630bf2f4ac0079701ed3187ca21f47858cca489c8\": container with ID starting with 55b5aa39b15b2800054928f630bf2f4ac0079701ed3187ca21f47858cca489c8 not found: ID does not exist" Sep 11 14:37:03.913086 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.913058 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j"] Sep 11 14:37:03.917855 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.917834 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-69a7cf-openshift-default-7b98fdbb88-kzv9j"] Sep 11 14:37:03.927490 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.927473 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e92780-d597-45c4-824d-9b453e24d070-istio-token\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:37:03.927548 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.927492 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-istio-envoy\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:37:03.927548 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.927501 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-workload-certs\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:37:03.927548 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.927508 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e92780-d597-45c4-824d-9b453e24d070-istio-podinfo\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:37:03.927548 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.927516 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e92780-d597-45c4-824d-9b453e24d070-istio-data\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:37:03.927548 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:03.927524 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p88c7\" (UniqueName: \"kubernetes.io/projected/e5e92780-d597-45c4-824d-9b453e24d070-kube-api-access-p88c7\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:37:05.631929 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:05.631890 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:05.912153 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:05.912121 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5e92780-d597-45c4-824d-9b453e24d070" path="/var/lib/kubelet/pods/e5e92780-d597-45c4-824d-9b453e24d070/volumes" Sep 11 14:37:06.568281 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:06.568242 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:11.002236 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:11.002204 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:14.164903 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:14.164817 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:14.312842 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:14.312805 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:14.431511 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:14.431402 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:15.283726 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:15.283690 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:15.437767 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:15.437731 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:15.574555 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:15.574476 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:16.569461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:16.569431 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:16.697536 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:16.697501 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:16.923850 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:16.923815 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:17.146711 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:17.146676 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:17.420610 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:17.420579 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:22.092520 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:22.092466 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:25.255761 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:25.255721 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:25.382493 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:25.382463 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:26.018320 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:26.018283 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:32.584685 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:32.584645 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:33.684192 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:33.684159 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q"] Sep 11 14:37:33.745945 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:33.745909 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:33.807475 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:33.807445 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d4444b-v5ps7"] Sep 11 14:37:33.807843 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:33.807831 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b1caf7a-9b03-441d-8d50-70805c09b911" containerName="main" Sep 11 14:37:33.807888 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:33.807844 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1caf7a-9b03-441d-8d50-70805c09b911" containerName="main" Sep 11 14:37:33.807888 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:33.807857 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e92780-d597-45c4-824d-9b453e24d070" containerName="istio-proxy" Sep 11 14:37:33.807888 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:33.807862 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e92780-d597-45c4-824d-9b453e24d070" containerName="istio-proxy" Sep 11 14:37:33.807888 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:33.807870 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5e92780-d597-45c4-824d-9b453e24d070" containerName="istio-proxy" Sep 11 14:37:33.807888 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:33.807876 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5e92780-d597-45c4-824d-9b453e24d070" containerName="istio-proxy" Sep 11 14:37:33.808052 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:33.807931 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5e92780-d597-45c4-824d-9b453e24d070" containerName="istio-proxy" Sep 11 14:37:33.808052 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:33.807940 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b1caf7a-9b03-441d-8d50-70805c09b911" containerName="main" Sep 11 14:37:33.811193 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:33.811179 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d4444b-v5ps7" Sep 11 14:37:33.819847 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:33.819828 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6975v\"" Sep 11 14:37:33.819967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:33.819827 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:37:33.837647 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:33.837623 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d4444b-v5ps7"] Sep 11 14:37:33.894612 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:33.894583 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bad680-40ec-4cdb-b5c4-bd34eb78f864-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d4444b-v5ps7\" (UID: \"d7bad680-40ec-4cdb-b5c4-bd34eb78f864\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d4444b-v5ps7" Sep 11 14:37:33.995894 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:33.995807 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bad680-40ec-4cdb-b5c4-bd34eb78f864-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d4444b-v5ps7\" (UID: \"d7bad680-40ec-4cdb-b5c4-bd34eb78f864\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d4444b-v5ps7" Sep 11 14:37:33.996441 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:33.996403 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bad680-40ec-4cdb-b5c4-bd34eb78f864-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d4444b-v5ps7\" (UID: \"d7bad680-40ec-4cdb-b5c4-bd34eb78f864\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d4444b-v5ps7" Sep 11 14:37:34.120941 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:34.120908 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d4444b-v5ps7" Sep 11 14:37:34.340377 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:34.340271 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:34.421344 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:34.421318 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d4444b-v5ps7"] Sep 11 14:37:34.423526 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:37:34.423500 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7bad680_40ec_4cdb_b5c4_bd34eb78f864.slice/crio-e8bf63ee21d84113e48d4ad84ed3d927538aa1136a7c679737b1222aca9b11b9 WatchSource:0}: Error finding container e8bf63ee21d84113e48d4ad84ed3d927538aa1136a7c679737b1222aca9b11b9: Status 404 returned error can't find the container with id e8bf63ee21d84113e48d4ad84ed3d927538aa1136a7c679737b1222aca9b11b9 Sep 11 14:37:34.467727 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:34.467690 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:35.002398 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:35.002309 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:35.017580 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:35.017550 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d4444b-v5ps7" event={"ID":"d7bad680-40ec-4cdb-b5c4-bd34eb78f864","Type":"ContainerStarted","Data":"daad23456631abc835fac1561575ddd82f0e49d4fab340c9d4f378079287e8a5"} Sep 11 14:37:35.017735 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:35.017605 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d4444b-v5ps7" event={"ID":"d7bad680-40ec-4cdb-b5c4-bd34eb78f864","Type":"ContainerStarted","Data":"e8bf63ee21d84113e48d4ad84ed3d927538aa1136a7c679737b1222aca9b11b9"} Sep 11 14:37:35.053158 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:35.053096 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d4444b-v5ps7" podStartSLOduration=2.053072038 podStartE2EDuration="2.053072038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:37:35.052237023 +0000 UTC m=+1365.666343390" watchObservedRunningTime="2026-09-11 14:37:35.053072038 +0000 UTC m=+1365.667178384" Sep 11 14:37:35.180436 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:35.180392 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:36.024680 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:36.024651 2553 generic.go:356] "Generic (PLEG): container finished" podID="d7bad680-40ec-4cdb-b5c4-bd34eb78f864" containerID="daad23456631abc835fac1561575ddd82f0e49d4fab340c9d4f378079287e8a5" exitCode=0 Sep 11 14:37:36.025046 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:36.024735 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d4444b-v5ps7" event={"ID":"d7bad680-40ec-4cdb-b5c4-bd34eb78f864","Type":"ContainerDied","Data":"daad23456631abc835fac1561575ddd82f0e49d4fab340c9d4f378079287e8a5"} Sep 11 14:37:36.138615 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:36.138582 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:37.165651 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:37.165621 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d4444b-v5ps7" Sep 11 14:37:37.227042 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:37.227015 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bad680-40ec-4cdb-b5c4-bd34eb78f864-maas-api-service-ca\") pod \"d7bad680-40ec-4cdb-b5c4-bd34eb78f864\" (UID: \"d7bad680-40ec-4cdb-b5c4-bd34eb78f864\") " Sep 11 14:37:37.227398 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:37.227371 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7bad680-40ec-4cdb-b5c4-bd34eb78f864-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d7bad680-40ec-4cdb-b5c4-bd34eb78f864" (UID: "d7bad680-40ec-4cdb-b5c4-bd34eb78f864"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:37:37.328481 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:37.328391 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7bad680-40ec-4cdb-b5c4-bd34eb78f864-maas-api-service-ca\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:37:37.572752 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:37.572720 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:38.037114 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:38.037085 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d4444b-v5ps7" Sep 11 14:37:38.037114 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:38.037100 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d4444b-v5ps7" event={"ID":"d7bad680-40ec-4cdb-b5c4-bd34eb78f864","Type":"ContainerDied","Data":"e8bf63ee21d84113e48d4ad84ed3d927538aa1136a7c679737b1222aca9b11b9"} Sep 11 14:37:38.037374 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:38.037143 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8bf63ee21d84113e48d4ad84ed3d927538aa1136a7c679737b1222aca9b11b9" Sep 11 14:37:38.411163 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:38.411130 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:38.518921 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:38.518890 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:39.209458 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:39.209410 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:40.511679 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:40.511641 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:40.608431 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:40.608388 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:40.755895 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:40.755858 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:41.128902 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.128867 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:41.377595 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.377563 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp"] Sep 11 14:37:41.377972 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.377960 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7bad680-40ec-4cdb-b5c4-bd34eb78f864" containerName="revoke-keys" Sep 11 14:37:41.378040 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.377974 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7bad680-40ec-4cdb-b5c4-bd34eb78f864" containerName="revoke-keys" Sep 11 14:37:41.378086 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.378044 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7bad680-40ec-4cdb-b5c4-bd34eb78f864" containerName="revoke-keys" Sep 11 14:37:41.378086 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.378054 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5e92780-d597-45c4-824d-9b453e24d070" containerName="istio-proxy" Sep 11 14:37:41.381251 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.381196 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp" Sep 11 14:37:41.387944 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.387763 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-210be13f-dockercfg-rss6q\"" Sep 11 14:37:41.388225 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.388205 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-210be13f\"" Sep 11 14:37:41.398509 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.398484 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp"] Sep 11 14:37:41.402248 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.402220 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:41.402841 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.402824 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s"] Sep 11 14:37:41.406300 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.406284 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s" Sep 11 14:37:41.430745 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.430714 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s"] Sep 11 14:37:41.464712 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.464684 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e70b02e-5f2a-45ef-bfe6-b50846475be9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp\" (UID: \"4e70b02e-5f2a-45ef-bfe6-b50846475be9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp" Sep 11 14:37:41.464887 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.464725 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xfqq\" (UniqueName: \"kubernetes.io/projected/4e70b02e-5f2a-45ef-bfe6-b50846475be9-kube-api-access-7xfqq\") pod \"payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp\" (UID: \"4e70b02e-5f2a-45ef-bfe6-b50846475be9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp" Sep 11 14:37:41.464969 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.464890 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvbt6\" (UniqueName: \"kubernetes.io/projected/2db78325-3261-4074-b8a8-538272cb67f4-kube-api-access-nvbt6\") pod \"payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s\" (UID: \"2db78325-3261-4074-b8a8-538272cb67f4\") " pod="openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s" Sep 11 14:37:41.464969 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.464941 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2db78325-3261-4074-b8a8-538272cb67f4-plugins-config-volume\") pod \"payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s\" (UID: \"2db78325-3261-4074-b8a8-538272cb67f4\") " pod="openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s" Sep 11 14:37:41.566232 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.566200 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e70b02e-5f2a-45ef-bfe6-b50846475be9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp\" (UID: \"4e70b02e-5f2a-45ef-bfe6-b50846475be9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp" Sep 11 14:37:41.566660 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.566246 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xfqq\" (UniqueName: \"kubernetes.io/projected/4e70b02e-5f2a-45ef-bfe6-b50846475be9-kube-api-access-7xfqq\") pod \"payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp\" (UID: \"4e70b02e-5f2a-45ef-bfe6-b50846475be9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp" Sep 11 14:37:41.566660 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.566312 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvbt6\" (UniqueName: \"kubernetes.io/projected/2db78325-3261-4074-b8a8-538272cb67f4-kube-api-access-nvbt6\") pod \"payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s\" (UID: \"2db78325-3261-4074-b8a8-538272cb67f4\") " pod="openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s" Sep 11 14:37:41.566660 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.566335 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2db78325-3261-4074-b8a8-538272cb67f4-plugins-config-volume\") pod \"payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s\" (UID: \"2db78325-3261-4074-b8a8-538272cb67f4\") " pod="openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s" Sep 11 14:37:41.567087 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.567068 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2db78325-3261-4074-b8a8-538272cb67f4-plugins-config-volume\") pod \"payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s\" (UID: \"2db78325-3261-4074-b8a8-538272cb67f4\") " pod="openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s" Sep 11 14:37:41.567169 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.567147 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e70b02e-5f2a-45ef-bfe6-b50846475be9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp\" (UID: \"4e70b02e-5f2a-45ef-bfe6-b50846475be9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp" Sep 11 14:37:41.578995 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.578966 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvbt6\" (UniqueName: \"kubernetes.io/projected/2db78325-3261-4074-b8a8-538272cb67f4-kube-api-access-nvbt6\") pod \"payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s\" (UID: \"2db78325-3261-4074-b8a8-538272cb67f4\") " pod="openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s" Sep 11 14:37:41.579118 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.578969 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xfqq\" (UniqueName: \"kubernetes.io/projected/4e70b02e-5f2a-45ef-bfe6-b50846475be9-kube-api-access-7xfqq\") pod \"payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp\" (UID: \"4e70b02e-5f2a-45ef-bfe6-b50846475be9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp" Sep 11 14:37:41.692292 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.692253 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp" Sep 11 14:37:41.715441 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.715393 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s" Sep 11 14:37:41.764294 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.764195 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:41.995248 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:41.995222 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp"] Sep 11 14:37:41.995968 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:37:41.995944 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e70b02e_5f2a_45ef_bfe6_b50846475be9.slice/crio-ff061e323637290765d3379c433970ba8e97fec693d6e6a6984bfbf61bff2440 WatchSource:0}: Error finding container ff061e323637290765d3379c433970ba8e97fec693d6e6a6984bfbf61bff2440: Status 404 returned error can't find the container with id ff061e323637290765d3379c433970ba8e97fec693d6e6a6984bfbf61bff2440 Sep 11 14:37:42.017483 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:42.017459 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s"] Sep 11 14:37:42.026499 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:37:42.026474 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db78325_3261_4074_b8a8_538272cb67f4.slice/crio-ab6c0f738d0da9cab405f26151d82f1e78967d0c9d60e22b9505931d551633d0 WatchSource:0}: Error finding container ab6c0f738d0da9cab405f26151d82f1e78967d0c9d60e22b9505931d551633d0: Status 404 returned error can't find the container with id ab6c0f738d0da9cab405f26151d82f1e78967d0c9d60e22b9505931d551633d0 Sep 11 14:37:42.058235 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:42.058206 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s" event={"ID":"2db78325-3261-4074-b8a8-538272cb67f4","Type":"ContainerStarted","Data":"ab6c0f738d0da9cab405f26151d82f1e78967d0c9d60e22b9505931d551633d0"} Sep 11 14:37:42.059857 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:42.059834 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp" event={"ID":"4e70b02e-5f2a-45ef-bfe6-b50846475be9","Type":"ContainerStarted","Data":"ff061e323637290765d3379c433970ba8e97fec693d6e6a6984bfbf61bff2440"} Sep 11 14:37:42.969337 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:42.969275 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:43.066930 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:43.066891 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s" event={"ID":"2db78325-3261-4074-b8a8-538272cb67f4","Type":"ContainerStarted","Data":"e1fedbbfdb1d5e44f983edc92b2f7f16b548aca9da6d27d5cd148f0340ac0a26"} Sep 11 14:37:43.067106 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:43.067069 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s" Sep 11 14:37:43.068522 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:43.068461 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp" event={"ID":"4e70b02e-5f2a-45ef-bfe6-b50846475be9","Type":"ContainerStarted","Data":"8d5c90beaf6b91abc738be576614954d5a9b01b3cbaf8b78b80d9353cc502d57"} Sep 11 14:37:43.068628 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:43.068550 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp" Sep 11 14:37:43.177867 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:43.177816 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp" podStartSLOduration=2.177799862 podStartE2EDuration="2.177799862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:37:43.176301529 +0000 UTC m=+1373.790407876" watchObservedRunningTime="2026-09-11 14:37:43.177799862 +0000 UTC m=+1373.791906206" Sep 11 14:37:43.179521 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:43.179490 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s" podStartSLOduration=2.179481001 podStartE2EDuration="2.179481001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:37:43.116874828 +0000 UTC m=+1373.730981171" watchObservedRunningTime="2026-09-11 14:37:43.179481001 +0000 UTC m=+1373.793587345" Sep 11 14:37:45.047965 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:45.047925 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:45.955208 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:45.955163 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:46.043139 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.043108 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7"] Sep 11 14:37:46.043342 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.043318 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7" podUID="12510bdd-4c7e-45b1-81bb-0289660fc613" containerName="payload-processing" containerID="cri-o://f58f605e596752848043fa5e822fcd1d603c6b5fbe4b397b4e7717722a89179f" gracePeriod=30 Sep 11 14:37:46.091402 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.091370 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv"] Sep 11 14:37:46.091791 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.091577 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv" podUID="fbaed836-34dd-4d8c-82c6-c6c23225c431" containerName="payload-pre-processing" containerID="cri-o://dd15c3f4e06dabd2f07e4f057390ac36c733537578ea4f3bc2283313d1721dd6" gracePeriod=30 Sep 11 14:37:46.111718 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.111695 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:46.219673 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.219636 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:46.343316 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.343289 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7" Sep 11 14:37:46.370861 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.370839 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv" Sep 11 14:37:46.414963 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.414934 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12510bdd-4c7e-45b1-81bb-0289660fc613-plugins-config-volume\") pod \"12510bdd-4c7e-45b1-81bb-0289660fc613\" (UID: \"12510bdd-4c7e-45b1-81bb-0289660fc613\") " Sep 11 14:37:46.415129 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.414974 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlbxc\" (UniqueName: \"kubernetes.io/projected/fbaed836-34dd-4d8c-82c6-c6c23225c431-kube-api-access-nlbxc\") pod \"fbaed836-34dd-4d8c-82c6-c6c23225c431\" (UID: \"fbaed836-34dd-4d8c-82c6-c6c23225c431\") " Sep 11 14:37:46.415129 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.415049 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbaed836-34dd-4d8c-82c6-c6c23225c431-plugins-config-volume\") pod \"fbaed836-34dd-4d8c-82c6-c6c23225c431\" (UID: \"fbaed836-34dd-4d8c-82c6-c6c23225c431\") " Sep 11 14:37:46.415129 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.415099 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhxjb\" (UniqueName: \"kubernetes.io/projected/12510bdd-4c7e-45b1-81bb-0289660fc613-kube-api-access-nhxjb\") pod \"12510bdd-4c7e-45b1-81bb-0289660fc613\" (UID: \"12510bdd-4c7e-45b1-81bb-0289660fc613\") " Sep 11 14:37:46.415395 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.415368 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12510bdd-4c7e-45b1-81bb-0289660fc613-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "12510bdd-4c7e-45b1-81bb-0289660fc613" (UID: "12510bdd-4c7e-45b1-81bb-0289660fc613"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:37:46.415524 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.415472 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbaed836-34dd-4d8c-82c6-c6c23225c431-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fbaed836-34dd-4d8c-82c6-c6c23225c431" (UID: "fbaed836-34dd-4d8c-82c6-c6c23225c431"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:37:46.417219 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.417201 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbaed836-34dd-4d8c-82c6-c6c23225c431-kube-api-access-nlbxc" (OuterVolumeSpecName: "kube-api-access-nlbxc") pod "fbaed836-34dd-4d8c-82c6-c6c23225c431" (UID: "fbaed836-34dd-4d8c-82c6-c6c23225c431"). InnerVolumeSpecName "kube-api-access-nlbxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:37:46.417285 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.417226 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12510bdd-4c7e-45b1-81bb-0289660fc613-kube-api-access-nhxjb" (OuterVolumeSpecName: "kube-api-access-nhxjb") pod "12510bdd-4c7e-45b1-81bb-0289660fc613" (UID: "12510bdd-4c7e-45b1-81bb-0289660fc613"). InnerVolumeSpecName "kube-api-access-nhxjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:37:46.493635 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.493548 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:46.516256 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.516223 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12510bdd-4c7e-45b1-81bb-0289660fc613-plugins-config-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:37:46.516256 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.516250 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlbxc\" (UniqueName: \"kubernetes.io/projected/fbaed836-34dd-4d8c-82c6-c6c23225c431-kube-api-access-nlbxc\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:37:46.516256 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.516259 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbaed836-34dd-4d8c-82c6-c6c23225c431-plugins-config-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:37:46.516497 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.516269 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhxjb\" (UniqueName: \"kubernetes.io/projected/12510bdd-4c7e-45b1-81bb-0289660fc613-kube-api-access-nhxjb\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:37:46.604756 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.604721 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d4444b-v5ps7"] Sep 11 14:37:46.617989 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.617960 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d4444b-v5ps7"] Sep 11 14:37:46.636574 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:46.636548 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:47.087250 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.087211 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:47.098773 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.098740 2553 generic.go:356] "Generic (PLEG): container finished" podID="fbaed836-34dd-4d8c-82c6-c6c23225c431" containerID="dd15c3f4e06dabd2f07e4f057390ac36c733537578ea4f3bc2283313d1721dd6" exitCode=0 Sep 11 14:37:47.099182 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.098773 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv" event={"ID":"fbaed836-34dd-4d8c-82c6-c6c23225c431","Type":"ContainerDied","Data":"dd15c3f4e06dabd2f07e4f057390ac36c733537578ea4f3bc2283313d1721dd6"} Sep 11 14:37:47.099182 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.098790 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv" Sep 11 14:37:47.099182 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.098854 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv" event={"ID":"fbaed836-34dd-4d8c-82c6-c6c23225c431","Type":"ContainerDied","Data":"50801c4fec7d360619e8c5040890dafdc5b2ac8bea39abf0984132e99d8bad6d"} Sep 11 14:37:47.099182 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.098876 2553 scope.go:117] "RemoveContainer" containerID="dd15c3f4e06dabd2f07e4f057390ac36c733537578ea4f3bc2283313d1721dd6" Sep 11 14:37:47.100320 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.100227 2553 generic.go:356] "Generic (PLEG): container finished" podID="12510bdd-4c7e-45b1-81bb-0289660fc613" containerID="f58f605e596752848043fa5e822fcd1d603c6b5fbe4b397b4e7717722a89179f" exitCode=0 Sep 11 14:37:47.100320 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.100266 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7" Sep 11 14:37:47.100320 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.100300 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7" event={"ID":"12510bdd-4c7e-45b1-81bb-0289660fc613","Type":"ContainerDied","Data":"f58f605e596752848043fa5e822fcd1d603c6b5fbe4b397b4e7717722a89179f"} Sep 11 14:37:47.100539 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.100332 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7" event={"ID":"12510bdd-4c7e-45b1-81bb-0289660fc613","Type":"ContainerDied","Data":"40d4e856a4297754336bded9b27eedc2bdda3685cc1631f3af30d18b5c0d6416"} Sep 11 14:37:47.107875 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.107729 2553 scope.go:117] "RemoveContainer" containerID="dd15c3f4e06dabd2f07e4f057390ac36c733537578ea4f3bc2283313d1721dd6" Sep 11 14:37:47.108078 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:37:47.108057 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd15c3f4e06dabd2f07e4f057390ac36c733537578ea4f3bc2283313d1721dd6\": container with ID starting with dd15c3f4e06dabd2f07e4f057390ac36c733537578ea4f3bc2283313d1721dd6 not found: ID does not exist" containerID="dd15c3f4e06dabd2f07e4f057390ac36c733537578ea4f3bc2283313d1721dd6" Sep 11 14:37:47.108143 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.108085 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd15c3f4e06dabd2f07e4f057390ac36c733537578ea4f3bc2283313d1721dd6"} err="failed to get container status \"dd15c3f4e06dabd2f07e4f057390ac36c733537578ea4f3bc2283313d1721dd6\": rpc error: code = NotFound desc = could not find container \"dd15c3f4e06dabd2f07e4f057390ac36c733537578ea4f3bc2283313d1721dd6\": container with ID starting with dd15c3f4e06dabd2f07e4f057390ac36c733537578ea4f3bc2283313d1721dd6 not found: ID does not exist" Sep 11 14:37:47.108143 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.108102 2553 scope.go:117] "RemoveContainer" containerID="f58f605e596752848043fa5e822fcd1d603c6b5fbe4b397b4e7717722a89179f" Sep 11 14:37:47.115872 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.115854 2553 scope.go:117] "RemoveContainer" containerID="f58f605e596752848043fa5e822fcd1d603c6b5fbe4b397b4e7717722a89179f" Sep 11 14:37:47.116156 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:37:47.116138 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f58f605e596752848043fa5e822fcd1d603c6b5fbe4b397b4e7717722a89179f\": container with ID starting with f58f605e596752848043fa5e822fcd1d603c6b5fbe4b397b4e7717722a89179f not found: ID does not exist" containerID="f58f605e596752848043fa5e822fcd1d603c6b5fbe4b397b4e7717722a89179f" Sep 11 14:37:47.116224 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.116163 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f58f605e596752848043fa5e822fcd1d603c6b5fbe4b397b4e7717722a89179f"} err="failed to get container status \"f58f605e596752848043fa5e822fcd1d603c6b5fbe4b397b4e7717722a89179f\": rpc error: code = NotFound desc = could not find container \"f58f605e596752848043fa5e822fcd1d603c6b5fbe4b397b4e7717722a89179f\": container with ID starting with f58f605e596752848043fa5e822fcd1d603c6b5fbe4b397b4e7717722a89179f not found: ID does not exist" Sep 11 14:37:47.143983 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.143952 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7"] Sep 11 14:37:47.154111 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.154089 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d4444b-5b459bd469-tzvc7"] Sep 11 14:37:47.190706 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.190672 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv"] Sep 11 14:37:47.193714 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.193685 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d4444b-77b88c974b-rpqfv"] Sep 11 14:37:47.321529 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.321442 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:47.912779 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.912749 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12510bdd-4c7e-45b1-81bb-0289660fc613" path="/var/lib/kubelet/pods/12510bdd-4c7e-45b1-81bb-0289660fc613/volumes" Sep 11 14:37:47.913084 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.913071 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7bad680-40ec-4cdb-b5c4-bd34eb78f864" path="/var/lib/kubelet/pods/d7bad680-40ec-4cdb-b5c4-bd34eb78f864/volumes" Sep 11 14:37:47.913376 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:47.913364 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbaed836-34dd-4d8c-82c6-c6c23225c431" path="/var/lib/kubelet/pods/fbaed836-34dd-4d8c-82c6-c6c23225c431/volumes" Sep 11 14:37:48.611399 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:48.611365 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:48.702951 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:48.702907 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" podUID="153b89ca-92c3-4562-b984-97c8af35ac2b" containerName="main" containerID="cri-o://57e4214f6d23a3c0dceac038d195977b9b2695d7fdf9a8f852384528e999ec99" gracePeriod=45 Sep 11 14:37:48.963025 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:48.963000 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:37:49.038877 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.038845 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-tmp-dir\") pod \"153b89ca-92c3-4562-b984-97c8af35ac2b\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " Sep 11 14:37:49.039060 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.038890 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfj8v\" (UniqueName: \"kubernetes.io/projected/153b89ca-92c3-4562-b984-97c8af35ac2b-kube-api-access-pfj8v\") pod \"153b89ca-92c3-4562-b984-97c8af35ac2b\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " Sep 11 14:37:49.039060 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.038942 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-model-cache\") pod \"153b89ca-92c3-4562-b984-97c8af35ac2b\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " Sep 11 14:37:49.039060 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.038965 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-home\") pod \"153b89ca-92c3-4562-b984-97c8af35ac2b\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " Sep 11 14:37:49.039241 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.039068 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-dshm\") pod \"153b89ca-92c3-4562-b984-97c8af35ac2b\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " Sep 11 14:37:49.039241 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.039135 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "153b89ca-92c3-4562-b984-97c8af35ac2b" (UID: "153b89ca-92c3-4562-b984-97c8af35ac2b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:37:49.039241 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.039155 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/153b89ca-92c3-4562-b984-97c8af35ac2b-tls-certs\") pod \"153b89ca-92c3-4562-b984-97c8af35ac2b\" (UID: \"153b89ca-92c3-4562-b984-97c8af35ac2b\") " Sep 11 14:37:49.039241 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.039164 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-home" (OuterVolumeSpecName: "home") pod "153b89ca-92c3-4562-b984-97c8af35ac2b" (UID: "153b89ca-92c3-4562-b984-97c8af35ac2b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:37:49.039435 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.039249 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-model-cache" (OuterVolumeSpecName: "model-cache") pod "153b89ca-92c3-4562-b984-97c8af35ac2b" (UID: "153b89ca-92c3-4562-b984-97c8af35ac2b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:37:49.039484 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.039433 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-tmp-dir\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:37:49.039484 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.039447 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-model-cache\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:37:49.039484 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.039457 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-home\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:37:49.041265 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.041233 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/153b89ca-92c3-4562-b984-97c8af35ac2b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "153b89ca-92c3-4562-b984-97c8af35ac2b" (UID: "153b89ca-92c3-4562-b984-97c8af35ac2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:37:49.041265 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.041254 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-dshm" (OuterVolumeSpecName: "dshm") pod "153b89ca-92c3-4562-b984-97c8af35ac2b" (UID: "153b89ca-92c3-4562-b984-97c8af35ac2b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:37:49.041403 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.041317 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153b89ca-92c3-4562-b984-97c8af35ac2b-kube-api-access-pfj8v" (OuterVolumeSpecName: "kube-api-access-pfj8v") pod "153b89ca-92c3-4562-b984-97c8af35ac2b" (UID: "153b89ca-92c3-4562-b984-97c8af35ac2b"). InnerVolumeSpecName "kube-api-access-pfj8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:37:49.113707 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.113673 2553 generic.go:356] "Generic (PLEG): container finished" podID="153b89ca-92c3-4562-b984-97c8af35ac2b" containerID="57e4214f6d23a3c0dceac038d195977b9b2695d7fdf9a8f852384528e999ec99" exitCode=0 Sep 11 14:37:49.113869 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.113715 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" Sep 11 14:37:49.113869 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.113759 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" event={"ID":"153b89ca-92c3-4562-b984-97c8af35ac2b","Type":"ContainerDied","Data":"57e4214f6d23a3c0dceac038d195977b9b2695d7fdf9a8f852384528e999ec99"} Sep 11 14:37:49.113869 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.113806 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q" event={"ID":"153b89ca-92c3-4562-b984-97c8af35ac2b","Type":"ContainerDied","Data":"27a0e37b803e90bde4400d9f7222fde0505e5647a7ec1fc7402c89232ee7955f"} Sep 11 14:37:49.113869 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.113827 2553 scope.go:117] "RemoveContainer" containerID="57e4214f6d23a3c0dceac038d195977b9b2695d7fdf9a8f852384528e999ec99" Sep 11 14:37:49.122463 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.122442 2553 scope.go:117] "RemoveContainer" containerID="57e4214f6d23a3c0dceac038d195977b9b2695d7fdf9a8f852384528e999ec99" Sep 11 14:37:49.122729 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:37:49.122712 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57e4214f6d23a3c0dceac038d195977b9b2695d7fdf9a8f852384528e999ec99\": container with ID starting with 57e4214f6d23a3c0dceac038d195977b9b2695d7fdf9a8f852384528e999ec99 not found: ID does not exist" containerID="57e4214f6d23a3c0dceac038d195977b9b2695d7fdf9a8f852384528e999ec99" Sep 11 14:37:49.122786 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.122739 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57e4214f6d23a3c0dceac038d195977b9b2695d7fdf9a8f852384528e999ec99"} err="failed to get container status \"57e4214f6d23a3c0dceac038d195977b9b2695d7fdf9a8f852384528e999ec99\": rpc error: code = NotFound desc = could not find container \"57e4214f6d23a3c0dceac038d195977b9b2695d7fdf9a8f852384528e999ec99\": container with ID starting with 57e4214f6d23a3c0dceac038d195977b9b2695d7fdf9a8f852384528e999ec99 not found: ID does not exist" Sep 11 14:37:49.140589 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.140562 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfj8v\" (UniqueName: \"kubernetes.io/projected/153b89ca-92c3-4562-b984-97c8af35ac2b-kube-api-access-pfj8v\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:37:49.140589 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.140585 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/153b89ca-92c3-4562-b984-97c8af35ac2b-dshm\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:37:49.140746 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.140595 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/153b89ca-92c3-4562-b984-97c8af35ac2b-tls-certs\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:37:49.141067 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.141049 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q"] Sep 11 14:37:49.148628 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.148601 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5b0917/test-model-5b0917-kserve-7865c64589-kwc2q"] Sep 11 14:37:49.857819 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.857785 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:49.914098 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:49.914068 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="153b89ca-92c3-4562-b984-97c8af35ac2b" path="/var/lib/kubelet/pods/153b89ca-92c3-4562-b984-97c8af35ac2b/volumes" Sep 11 14:37:51.665551 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:51.665513 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:52.810345 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:52.810311 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:53.062437 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:53.062314 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:53.859310 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:53.859266 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:54.076236 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:54.076206 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp" Sep 11 14:37:54.076434 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:54.076280 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s" Sep 11 14:37:54.135532 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:54.135456 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:54.394164 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:54.394133 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:54.729157 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:54.729052 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:56.885271 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:56.885238 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:56.990747 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:56.990711 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:57.223218 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:57.223188 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:57.647353 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:57.647321 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b0917-89zcw"] Sep 11 14:37:57.647763 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:57.647750 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbaed836-34dd-4d8c-82c6-c6c23225c431" containerName="payload-pre-processing" Sep 11 14:37:57.647825 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:57.647764 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbaed836-34dd-4d8c-82c6-c6c23225c431" containerName="payload-pre-processing" Sep 11 14:37:57.647825 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:57.647773 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="153b89ca-92c3-4562-b984-97c8af35ac2b" containerName="main" Sep 11 14:37:57.647825 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:57.647778 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="153b89ca-92c3-4562-b984-97c8af35ac2b" containerName="main" Sep 11 14:37:57.647825 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:57.647796 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12510bdd-4c7e-45b1-81bb-0289660fc613" containerName="payload-processing" Sep 11 14:37:57.647825 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:57.647802 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="12510bdd-4c7e-45b1-81bb-0289660fc613" containerName="payload-processing" Sep 11 14:37:57.647977 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:57.647864 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbaed836-34dd-4d8c-82c6-c6c23225c431" containerName="payload-pre-processing" Sep 11 14:37:57.647977 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:57.647873 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="12510bdd-4c7e-45b1-81bb-0289660fc613" containerName="payload-processing" Sep 11 14:37:57.647977 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:57.647882 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="153b89ca-92c3-4562-b984-97c8af35ac2b" containerName="main" Sep 11 14:37:57.652207 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:57.652188 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b0917-89zcw" Sep 11 14:37:57.656601 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:57.656584 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:37:57.657116 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:57.657099 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6975v\"" Sep 11 14:37:57.670146 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:57.670116 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b0917-89zcw"] Sep 11 14:37:57.720386 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:57.720355 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43991194-ed2f-45d5-9adb-d4ca32680812-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5b0917-89zcw\" (UID: \"43991194-ed2f-45d5-9adb-d4ca32680812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b0917-89zcw" Sep 11 14:37:57.820881 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:57.820846 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43991194-ed2f-45d5-9adb-d4ca32680812-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5b0917-89zcw\" (UID: \"43991194-ed2f-45d5-9adb-d4ca32680812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b0917-89zcw" Sep 11 14:37:57.821449 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:57.821428 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43991194-ed2f-45d5-9adb-d4ca32680812-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5b0917-89zcw\" (UID: \"43991194-ed2f-45d5-9adb-d4ca32680812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b0917-89zcw" Sep 11 14:37:57.962023 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:57.961938 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b0917-89zcw" Sep 11 14:37:58.194674 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:58.194635 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:37:58.281626 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:58.281534 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b0917-89zcw"] Sep 11 14:37:58.286022 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:37:58.285994 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43991194_ed2f_45d5_9adb_d4ca32680812.slice/crio-3d900a6e7907464f8aa143e16e145819eaac885d6155b6cb552a4d25d29dc169 WatchSource:0}: Error finding container 3d900a6e7907464f8aa143e16e145819eaac885d6155b6cb552a4d25d29dc169: Status 404 returned error can't find the container with id 3d900a6e7907464f8aa143e16e145819eaac885d6155b6cb552a4d25d29dc169 Sep 11 14:37:59.165582 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:59.165549 2553 generic.go:356] "Generic (PLEG): container finished" podID="43991194-ed2f-45d5-9adb-d4ca32680812" containerID="fbd71b86c7476df4755dab457e0f89ecc37158b5af0dccabd56d272a86f7ffd1" exitCode=0 Sep 11 14:37:59.165942 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:59.165639 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b0917-89zcw" event={"ID":"43991194-ed2f-45d5-9adb-d4ca32680812","Type":"ContainerDied","Data":"fbd71b86c7476df4755dab457e0f89ecc37158b5af0dccabd56d272a86f7ffd1"} Sep 11 14:37:59.165942 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:59.165687 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b0917-89zcw" event={"ID":"43991194-ed2f-45d5-9adb-d4ca32680812","Type":"ContainerStarted","Data":"3d900a6e7907464f8aa143e16e145819eaac885d6155b6cb552a4d25d29dc169"} Sep 11 14:37:59.707989 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:37:59.707951 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:00.300632 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:00.300607 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b0917-89zcw" Sep 11 14:38:00.444874 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:00.444784 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43991194-ed2f-45d5-9adb-d4ca32680812-maas-api-service-ca\") pod \"43991194-ed2f-45d5-9adb-d4ca32680812\" (UID: \"43991194-ed2f-45d5-9adb-d4ca32680812\") " Sep 11 14:38:00.445156 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:00.445136 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43991194-ed2f-45d5-9adb-d4ca32680812-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "43991194-ed2f-45d5-9adb-d4ca32680812" (UID: "43991194-ed2f-45d5-9adb-d4ca32680812"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:38:00.545520 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:00.545490 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43991194-ed2f-45d5-9adb-d4ca32680812-maas-api-service-ca\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:38:00.998291 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:00.998260 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:01.176907 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:01.176877 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b0917-89zcw" Sep 11 14:38:01.177056 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:01.176877 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b0917-89zcw" event={"ID":"43991194-ed2f-45d5-9adb-d4ca32680812","Type":"ContainerDied","Data":"3d900a6e7907464f8aa143e16e145819eaac885d6155b6cb552a4d25d29dc169"} Sep 11 14:38:01.177056 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:01.177006 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d900a6e7907464f8aa143e16e145819eaac885d6155b6cb552a4d25d29dc169" Sep 11 14:38:02.129680 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:02.129639 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:02.270244 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:02.270212 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:02.836761 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:02.836729 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:02.943618 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:02.943576 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:03.573698 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:03.573658 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:04.971154 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:04.971124 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:06.259645 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:06.259614 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:07.449545 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:07.449509 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:08.374675 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:08.374637 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:08.476554 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:08.476520 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8"] Sep 11 14:38:08.477061 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:08.476746 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8" podUID="d25ca92e-bd88-4c25-a601-16fc41c3fcd3" containerName="payload-pre-processing" containerID="cri-o://0e972de7a56baaf9d0ae7349d193a8fbb33d3cd1701812bead46f26cea8c265c" gracePeriod=30 Sep 11 14:38:08.495612 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:08.495571 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:08.597532 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:08.597506 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:08.699537 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:08.699499 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:08.731624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:08.731600 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8" Sep 11 14:38:08.796882 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:08.796848 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b0917-89zcw"] Sep 11 14:38:08.807648 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:08.807610 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5b0917-89zcw"] Sep 11 14:38:08.823087 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:08.823059 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwf4w\" (UniqueName: \"kubernetes.io/projected/d25ca92e-bd88-4c25-a601-16fc41c3fcd3-kube-api-access-hwf4w\") pod \"d25ca92e-bd88-4c25-a601-16fc41c3fcd3\" (UID: \"d25ca92e-bd88-4c25-a601-16fc41c3fcd3\") " Sep 11 14:38:08.823232 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:08.823143 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25ca92e-bd88-4c25-a601-16fc41c3fcd3-plugins-config-volume\") pod \"d25ca92e-bd88-4c25-a601-16fc41c3fcd3\" (UID: \"d25ca92e-bd88-4c25-a601-16fc41c3fcd3\") " Sep 11 14:38:08.823543 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:08.823520 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d25ca92e-bd88-4c25-a601-16fc41c3fcd3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d25ca92e-bd88-4c25-a601-16fc41c3fcd3" (UID: "d25ca92e-bd88-4c25-a601-16fc41c3fcd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:38:08.825316 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:08.825293 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25ca92e-bd88-4c25-a601-16fc41c3fcd3-kube-api-access-hwf4w" (OuterVolumeSpecName: "kube-api-access-hwf4w") pod "d25ca92e-bd88-4c25-a601-16fc41c3fcd3" (UID: "d25ca92e-bd88-4c25-a601-16fc41c3fcd3"). InnerVolumeSpecName "kube-api-access-hwf4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:38:08.872377 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:08.872350 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:08.923742 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:08.923664 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25ca92e-bd88-4c25-a601-16fc41c3fcd3-plugins-config-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:38:08.923742 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:08.923689 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwf4w\" (UniqueName: \"kubernetes.io/projected/d25ca92e-bd88-4c25-a601-16fc41c3fcd3-kube-api-access-hwf4w\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:38:09.219158 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:09.219071 2553 generic.go:356] "Generic (PLEG): container finished" podID="d25ca92e-bd88-4c25-a601-16fc41c3fcd3" containerID="0e972de7a56baaf9d0ae7349d193a8fbb33d3cd1701812bead46f26cea8c265c" exitCode=0 Sep 11 14:38:09.219158 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:09.219112 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8" Sep 11 14:38:09.219158 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:09.219141 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8" event={"ID":"d25ca92e-bd88-4c25-a601-16fc41c3fcd3","Type":"ContainerDied","Data":"0e972de7a56baaf9d0ae7349d193a8fbb33d3cd1701812bead46f26cea8c265c"} Sep 11 14:38:09.219404 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:09.219187 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8" event={"ID":"d25ca92e-bd88-4c25-a601-16fc41c3fcd3","Type":"ContainerDied","Data":"7cab169311032a7311ad72cd2a5c98e4e734e18102d59d7ddfb044934cfae9d4"} Sep 11 14:38:09.219404 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:09.219213 2553 scope.go:117] "RemoveContainer" containerID="0e972de7a56baaf9d0ae7349d193a8fbb33d3cd1701812bead46f26cea8c265c" Sep 11 14:38:09.227701 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:09.227685 2553 scope.go:117] "RemoveContainer" containerID="0e972de7a56baaf9d0ae7349d193a8fbb33d3cd1701812bead46f26cea8c265c" Sep 11 14:38:09.227964 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:38:09.227949 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e972de7a56baaf9d0ae7349d193a8fbb33d3cd1701812bead46f26cea8c265c\": container with ID starting with 0e972de7a56baaf9d0ae7349d193a8fbb33d3cd1701812bead46f26cea8c265c not found: ID does not exist" containerID="0e972de7a56baaf9d0ae7349d193a8fbb33d3cd1701812bead46f26cea8c265c" Sep 11 14:38:09.228009 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:09.227972 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e972de7a56baaf9d0ae7349d193a8fbb33d3cd1701812bead46f26cea8c265c"} err="failed to get container status \"0e972de7a56baaf9d0ae7349d193a8fbb33d3cd1701812bead46f26cea8c265c\": rpc error: code = NotFound desc = could not find container \"0e972de7a56baaf9d0ae7349d193a8fbb33d3cd1701812bead46f26cea8c265c\": container with ID starting with 0e972de7a56baaf9d0ae7349d193a8fbb33d3cd1701812bead46f26cea8c265c not found: ID does not exist" Sep 11 14:38:09.258525 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:09.258495 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8"] Sep 11 14:38:09.264058 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:09.264028 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5b0917-77b4c87b8b-qgxl8"] Sep 11 14:38:09.287498 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:09.287476 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:09.912746 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:09.912713 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43991194-ed2f-45d5-9adb-d4ca32680812" path="/var/lib/kubelet/pods/43991194-ed2f-45d5-9adb-d4ca32680812/volumes" Sep 11 14:38:09.913098 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:09.913083 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d25ca92e-bd88-4c25-a601-16fc41c3fcd3" path="/var/lib/kubelet/pods/d25ca92e-bd88-4c25-a601-16fc41c3fcd3/volumes" Sep 11 14:38:10.217777 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:10.217672 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:11.056227 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.056195 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:11.111110 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.111071 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s"] Sep 11 14:38:11.111373 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.111341 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s" podUID="2db78325-3261-4074-b8a8-538272cb67f4" containerName="payload-processing" containerID="cri-o://e1fedbbfdb1d5e44f983edc92b2f7f16b548aca9da6d27d5cd148f0340ac0a26" gracePeriod=30 Sep 11 14:38:11.131239 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.131215 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp"] Sep 11 14:38:11.131450 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.131404 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp" podUID="4e70b02e-5f2a-45ef-bfe6-b50846475be9" containerName="payload-pre-processing" containerID="cri-o://8d5c90beaf6b91abc738be576614954d5a9b01b3cbaf8b78b80d9353cc502d57" gracePeriod=30 Sep 11 14:38:11.195404 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.195371 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:11.230797 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.230761 2553 generic.go:356] "Generic (PLEG): container finished" podID="2db78325-3261-4074-b8a8-538272cb67f4" containerID="e1fedbbfdb1d5e44f983edc92b2f7f16b548aca9da6d27d5cd148f0340ac0a26" exitCode=0 Sep 11 14:38:11.230935 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.230833 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s" event={"ID":"2db78325-3261-4074-b8a8-538272cb67f4","Type":"ContainerDied","Data":"e1fedbbfdb1d5e44f983edc92b2f7f16b548aca9da6d27d5cd148f0340ac0a26"} Sep 11 14:38:11.333176 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.333138 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:11.404259 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.404236 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s" Sep 11 14:38:11.407453 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.407430 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp" Sep 11 14:38:11.474200 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.474170 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:11.548624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.548593 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e70b02e-5f2a-45ef-bfe6-b50846475be9-plugins-config-volume\") pod \"4e70b02e-5f2a-45ef-bfe6-b50846475be9\" (UID: \"4e70b02e-5f2a-45ef-bfe6-b50846475be9\") " Sep 11 14:38:11.548797 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.548716 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvbt6\" (UniqueName: \"kubernetes.io/projected/2db78325-3261-4074-b8a8-538272cb67f4-kube-api-access-nvbt6\") pod \"2db78325-3261-4074-b8a8-538272cb67f4\" (UID: \"2db78325-3261-4074-b8a8-538272cb67f4\") " Sep 11 14:38:11.548797 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.548751 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xfqq\" (UniqueName: \"kubernetes.io/projected/4e70b02e-5f2a-45ef-bfe6-b50846475be9-kube-api-access-7xfqq\") pod \"4e70b02e-5f2a-45ef-bfe6-b50846475be9\" (UID: \"4e70b02e-5f2a-45ef-bfe6-b50846475be9\") " Sep 11 14:38:11.548945 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.548807 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2db78325-3261-4074-b8a8-538272cb67f4-plugins-config-volume\") pod \"2db78325-3261-4074-b8a8-538272cb67f4\" (UID: \"2db78325-3261-4074-b8a8-538272cb67f4\") " Sep 11 14:38:11.549035 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.549009 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e70b02e-5f2a-45ef-bfe6-b50846475be9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e70b02e-5f2a-45ef-bfe6-b50846475be9" (UID: "4e70b02e-5f2a-45ef-bfe6-b50846475be9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:38:11.549166 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.549150 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e70b02e-5f2a-45ef-bfe6-b50846475be9-plugins-config-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:38:11.549288 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.549255 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db78325-3261-4074-b8a8-538272cb67f4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2db78325-3261-4074-b8a8-538272cb67f4" (UID: "2db78325-3261-4074-b8a8-538272cb67f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:38:11.550868 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.550845 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e70b02e-5f2a-45ef-bfe6-b50846475be9-kube-api-access-7xfqq" (OuterVolumeSpecName: "kube-api-access-7xfqq") pod "4e70b02e-5f2a-45ef-bfe6-b50846475be9" (UID: "4e70b02e-5f2a-45ef-bfe6-b50846475be9"). InnerVolumeSpecName "kube-api-access-7xfqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:38:11.550868 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.550855 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db78325-3261-4074-b8a8-538272cb67f4-kube-api-access-nvbt6" (OuterVolumeSpecName: "kube-api-access-nvbt6") pod "2db78325-3261-4074-b8a8-538272cb67f4" (UID: "2db78325-3261-4074-b8a8-538272cb67f4"). InnerVolumeSpecName "kube-api-access-nvbt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:38:11.649575 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.649537 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:11.649728 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.649707 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvbt6\" (UniqueName: \"kubernetes.io/projected/2db78325-3261-4074-b8a8-538272cb67f4-kube-api-access-nvbt6\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:38:11.649785 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.649730 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xfqq\" (UniqueName: \"kubernetes.io/projected/4e70b02e-5f2a-45ef-bfe6-b50846475be9-kube-api-access-7xfqq\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:38:11.649785 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.649745 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2db78325-3261-4074-b8a8-538272cb67f4-plugins-config-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:38:11.995786 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:11.995738 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:12.240959 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:12.240877 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s" Sep 11 14:38:12.240959 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:12.240878 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s" event={"ID":"2db78325-3261-4074-b8a8-538272cb67f4","Type":"ContainerDied","Data":"ab6c0f738d0da9cab405f26151d82f1e78967d0c9d60e22b9505931d551633d0"} Sep 11 14:38:12.241498 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:12.241014 2553 scope.go:117] "RemoveContainer" containerID="e1fedbbfdb1d5e44f983edc92b2f7f16b548aca9da6d27d5cd148f0340ac0a26" Sep 11 14:38:12.242799 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:12.242617 2553 generic.go:356] "Generic (PLEG): container finished" podID="4e70b02e-5f2a-45ef-bfe6-b50846475be9" containerID="8d5c90beaf6b91abc738be576614954d5a9b01b3cbaf8b78b80d9353cc502d57" exitCode=0 Sep 11 14:38:12.242799 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:12.242685 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp" event={"ID":"4e70b02e-5f2a-45ef-bfe6-b50846475be9","Type":"ContainerDied","Data":"8d5c90beaf6b91abc738be576614954d5a9b01b3cbaf8b78b80d9353cc502d57"} Sep 11 14:38:12.242799 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:12.242691 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp" Sep 11 14:38:12.242799 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:12.242719 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp" event={"ID":"4e70b02e-5f2a-45ef-bfe6-b50846475be9","Type":"ContainerDied","Data":"ff061e323637290765d3379c433970ba8e97fec693d6e6a6984bfbf61bff2440"} Sep 11 14:38:12.250530 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:12.250489 2553 scope.go:117] "RemoveContainer" containerID="8d5c90beaf6b91abc738be576614954d5a9b01b3cbaf8b78b80d9353cc502d57" Sep 11 14:38:12.258490 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:12.258468 2553 scope.go:117] "RemoveContainer" containerID="8d5c90beaf6b91abc738be576614954d5a9b01b3cbaf8b78b80d9353cc502d57" Sep 11 14:38:12.258829 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:38:12.258806 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d5c90beaf6b91abc738be576614954d5a9b01b3cbaf8b78b80d9353cc502d57\": container with ID starting with 8d5c90beaf6b91abc738be576614954d5a9b01b3cbaf8b78b80d9353cc502d57 not found: ID does not exist" containerID="8d5c90beaf6b91abc738be576614954d5a9b01b3cbaf8b78b80d9353cc502d57" Sep 11 14:38:12.258915 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:12.258842 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d5c90beaf6b91abc738be576614954d5a9b01b3cbaf8b78b80d9353cc502d57"} err="failed to get container status \"8d5c90beaf6b91abc738be576614954d5a9b01b3cbaf8b78b80d9353cc502d57\": rpc error: code = NotFound desc = could not find container \"8d5c90beaf6b91abc738be576614954d5a9b01b3cbaf8b78b80d9353cc502d57\": container with ID starting with 8d5c90beaf6b91abc738be576614954d5a9b01b3cbaf8b78b80d9353cc502d57 not found: ID does not exist" Sep 11 14:38:12.269776 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:12.269754 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp"] Sep 11 14:38:12.280503 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:12.280479 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-210be13f-7c646dc6d6-lv2wp"] Sep 11 14:38:12.297784 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:12.297759 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s"] Sep 11 14:38:12.302975 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:12.302945 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-210be13f-786d4d7c69-qc95s"] Sep 11 14:38:13.911865 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:13.911820 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2db78325-3261-4074-b8a8-538272cb67f4" path="/var/lib/kubelet/pods/2db78325-3261-4074-b8a8-538272cb67f4/volumes" Sep 11 14:38:13.912379 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:13.912363 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e70b02e-5f2a-45ef-bfe6-b50846475be9" path="/var/lib/kubelet/pods/4e70b02e-5f2a-45ef-bfe6-b50846475be9/volumes" Sep 11 14:38:16.384288 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:16.384250 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:16.563928 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:16.563883 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:16.734281 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:16.734239 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:17.412323 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:17.412287 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:17.840998 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:17.840907 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2"] Sep 11 14:38:17.841348 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:17.841330 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e70b02e-5f2a-45ef-bfe6-b50846475be9" containerName="payload-pre-processing" Sep 11 14:38:17.841458 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:17.841350 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e70b02e-5f2a-45ef-bfe6-b50846475be9" containerName="payload-pre-processing" Sep 11 14:38:17.841458 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:17.841370 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43991194-ed2f-45d5-9adb-d4ca32680812" containerName="revoke-keys" Sep 11 14:38:17.841458 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:17.841378 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="43991194-ed2f-45d5-9adb-d4ca32680812" containerName="revoke-keys" Sep 11 14:38:17.841458 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:17.841395 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d25ca92e-bd88-4c25-a601-16fc41c3fcd3" containerName="payload-pre-processing" Sep 11 14:38:17.841458 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:17.841403 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="d25ca92e-bd88-4c25-a601-16fc41c3fcd3" containerName="payload-pre-processing" Sep 11 14:38:17.841458 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:17.841438 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db78325-3261-4074-b8a8-538272cb67f4" containerName="payload-processing" Sep 11 14:38:17.841458 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:17.841448 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db78325-3261-4074-b8a8-538272cb67f4" containerName="payload-processing" Sep 11 14:38:17.841804 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:17.841556 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2db78325-3261-4074-b8a8-538272cb67f4" containerName="payload-processing" Sep 11 14:38:17.841804 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:17.841569 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="d25ca92e-bd88-4c25-a601-16fc41c3fcd3" containerName="payload-pre-processing" Sep 11 14:38:17.841804 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:17.841579 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e70b02e-5f2a-45ef-bfe6-b50846475be9" containerName="payload-pre-processing" Sep 11 14:38:17.841804 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:17.841588 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="43991194-ed2f-45d5-9adb-d4ca32680812" containerName="revoke-keys" Sep 11 14:38:17.845606 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:17.845585 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:17.850188 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:17.850165 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-8feb68-openshift-default-dockercfg-85j2r\"" Sep 11 14:38:17.850279 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:17.850225 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 14:38:17.893100 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:17.893069 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2"] Sep 11 14:38:17.902641 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:17.902610 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:18.007328 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.007289 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-workload-socket\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.007543 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.007348 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ee52bc-2034-4487-8030-93c61a828509-istio-podinfo\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.007543 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.007382 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-credential-socket\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.007543 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.007402 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-istio-data\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.007543 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.007436 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ee52bc-2034-4487-8030-93c61a828509-istiod-ca-cert\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.007543 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.007521 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfmgt\" (UniqueName: \"kubernetes.io/projected/b0ee52bc-2034-4487-8030-93c61a828509-kube-api-access-kfmgt\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.007786 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.007567 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ee52bc-2034-4487-8030-93c61a828509-istio-token\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.007786 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.007615 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-workload-certs\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.007786 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.007657 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-istio-envoy\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.108790 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.108687 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ee52bc-2034-4487-8030-93c61a828509-istio-podinfo\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.108790 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.108727 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-credential-socket\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.108790 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.108746 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-istio-data\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.108790 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.108761 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ee52bc-2034-4487-8030-93c61a828509-istiod-ca-cert\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.109146 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.108806 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfmgt\" (UniqueName: \"kubernetes.io/projected/b0ee52bc-2034-4487-8030-93c61a828509-kube-api-access-kfmgt\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.109146 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.108838 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ee52bc-2034-4487-8030-93c61a828509-istio-token\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.109146 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.108884 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-workload-certs\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.109146 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.108942 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-istio-envoy\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.109146 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.108983 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-workload-socket\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.109410 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.109177 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-credential-socket\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.109664 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.109639 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-workload-socket\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.109759 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.109669 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-istio-data\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.109759 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.109719 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-workload-certs\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.109874 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.109769 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ee52bc-2034-4487-8030-93c61a828509-istiod-ca-cert\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.111610 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.111584 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-istio-envoy\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.111750 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.111732 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ee52bc-2034-4487-8030-93c61a828509-istio-podinfo\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.130882 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.130853 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfmgt\" (UniqueName: \"kubernetes.io/projected/b0ee52bc-2034-4487-8030-93c61a828509-kube-api-access-kfmgt\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.130998 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.130915 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ee52bc-2034-4487-8030-93c61a828509-istio-token\") pod \"e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.155934 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.155884 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:18.453606 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.453565 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:18.457186 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.457158 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2"] Sep 11 14:38:18.460230 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:38:18.460203 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ee52bc_2034_4487_8030_93c61a828509.slice/crio-9d5464bcf73ab14a0dc5a91e8ac189f077dd909725883185899198d78e2646f2 WatchSource:0}: Error finding container 9d5464bcf73ab14a0dc5a91e8ac189f077dd909725883185899198d78e2646f2: Status 404 returned error can't find the container with id 9d5464bcf73ab14a0dc5a91e8ac189f077dd909725883185899198d78e2646f2 Sep 11 14:38:18.462446 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.462400 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:38:18.462525 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.462482 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:38:18.462525 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:18.462512 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:38:19.281003 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:19.280971 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" event={"ID":"b0ee52bc-2034-4487-8030-93c61a828509","Type":"ContainerStarted","Data":"9178072f3c2888db7ef81da9171b03edefa570451c2ef893a088048fd2ce9ee9"} Sep 11 14:38:19.281203 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:19.281023 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" event={"ID":"b0ee52bc-2034-4487-8030-93c61a828509","Type":"ContainerStarted","Data":"9d5464bcf73ab14a0dc5a91e8ac189f077dd909725883185899198d78e2646f2"} Sep 11 14:38:19.342752 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:19.342705 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" podStartSLOduration=2.34268943 podStartE2EDuration="2.34268943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:38:19.341406799 +0000 UTC m=+1409.955513144" watchObservedRunningTime="2026-09-11 14:38:19.34268943 +0000 UTC m=+1409.956795774" Sep 11 14:38:20.156286 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:20.156251 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:20.161067 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:20.161043 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:20.286212 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:20.286179 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:20.287130 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:20.287111 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:21.514086 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:21.514049 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:21.648860 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:21.648829 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:21.850077 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:21.849958 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:22.036257 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:22.036220 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:22.309918 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:22.309880 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:22.425124 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:22.425094 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:22.638571 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:22.638538 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:23.064880 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:23.064789 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:23.206399 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:23.206363 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:24.259228 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:24.259198 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:24.500841 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:24.500809 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:24.883229 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:24.883194 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z"] Sep 11 14:38:24.886532 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:24.886516 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z" Sep 11 14:38:24.890426 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:24.890388 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-8feb68-dockercfg-v4tcj\"" Sep 11 14:38:24.890676 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:24.890660 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-8feb68\"" Sep 11 14:38:24.965915 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:24.965882 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50abaebd-162a-414d-9e3e-9c84ef2d30c1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z\" (UID: \"50abaebd-162a-414d-9e3e-9c84ef2d30c1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z" Sep 11 14:38:24.966088 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:24.965928 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdhbk\" (UniqueName: \"kubernetes.io/projected/50abaebd-162a-414d-9e3e-9c84ef2d30c1-kube-api-access-sdhbk\") pod \"payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z\" (UID: \"50abaebd-162a-414d-9e3e-9c84ef2d30c1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z" Sep 11 14:38:24.987669 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:24.987640 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z"] Sep 11 14:38:25.066359 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:25.066323 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdhbk\" (UniqueName: \"kubernetes.io/projected/50abaebd-162a-414d-9e3e-9c84ef2d30c1-kube-api-access-sdhbk\") pod \"payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z\" (UID: \"50abaebd-162a-414d-9e3e-9c84ef2d30c1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z" Sep 11 14:38:25.066575 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:25.066433 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50abaebd-162a-414d-9e3e-9c84ef2d30c1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z\" (UID: \"50abaebd-162a-414d-9e3e-9c84ef2d30c1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z" Sep 11 14:38:25.067037 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:25.067019 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50abaebd-162a-414d-9e3e-9c84ef2d30c1-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z\" (UID: \"50abaebd-162a-414d-9e3e-9c84ef2d30c1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z" Sep 11 14:38:25.079172 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:25.079147 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdhbk\" (UniqueName: \"kubernetes.io/projected/50abaebd-162a-414d-9e3e-9c84ef2d30c1-kube-api-access-sdhbk\") pod \"payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z\" (UID: \"50abaebd-162a-414d-9e3e-9c84ef2d30c1\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z" Sep 11 14:38:25.197432 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:25.197377 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z" Sep 11 14:38:25.502579 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:25.502370 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z"] Sep 11 14:38:25.504884 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:38:25.504857 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50abaebd_162a_414d_9e3e_9c84ef2d30c1.slice/crio-f142bfa53f21641e438826c98d941f811c7298f6dc9b1ac5a8bd4a575ccf898d WatchSource:0}: Error finding container f142bfa53f21641e438826c98d941f811c7298f6dc9b1ac5a8bd4a575ccf898d: Status 404 returned error can't find the container with id f142bfa53f21641e438826c98d941f811c7298f6dc9b1ac5a8bd4a575ccf898d Sep 11 14:38:26.318104 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:26.318067 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z" event={"ID":"50abaebd-162a-414d-9e3e-9c84ef2d30c1","Type":"ContainerStarted","Data":"06d76460c26745d863d0164e0f335028fa7cbb81095dd0b6283fe0f8b9f1ed7a"} Sep 11 14:38:26.318288 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:26.318127 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z" event={"ID":"50abaebd-162a-414d-9e3e-9c84ef2d30c1","Type":"ContainerStarted","Data":"f142bfa53f21641e438826c98d941f811c7298f6dc9b1ac5a8bd4a575ccf898d"} Sep 11 14:38:26.318288 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:26.318179 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z" Sep 11 14:38:29.554681 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:29.554640 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:29.770038 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:29.769999 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:29.862334 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:29.862226 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:29.996981 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:29.996942 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:35.881654 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:35.880377 2553 patch_prober.go:28] interesting pod/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": read tcp 10.132.0.2:35758->10.132.0.62:15021: read: connection reset by peer" start-of-body= Sep 11 14:38:35.881654 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:35.880457 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" podUID="b0ee52bc-2034-4487-8030-93c61a828509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": read tcp 10.132.0.2:35758->10.132.0.62:15021: read: connection reset by peer" Sep 11 14:38:36.369997 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:36.369960 2553 generic.go:356] "Generic (PLEG): container finished" podID="b0ee52bc-2034-4487-8030-93c61a828509" containerID="9178072f3c2888db7ef81da9171b03edefa570451c2ef893a088048fd2ce9ee9" exitCode=0 Sep 11 14:38:36.370172 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:36.370036 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" event={"ID":"b0ee52bc-2034-4487-8030-93c61a828509","Type":"ContainerDied","Data":"9178072f3c2888db7ef81da9171b03edefa570451c2ef893a088048fd2ce9ee9"} Sep 11 14:38:36.370403 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:36.370390 2553 scope.go:117] "RemoveContainer" containerID="9178072f3c2888db7ef81da9171b03edefa570451c2ef893a088048fd2ce9ee9" Sep 11 14:38:36.371408 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:36.371378 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:38:36.371534 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:36.371477 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:38:36.371534 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:36.371529 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:38:37.326278 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:37.326249 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z" Sep 11 14:38:37.377141 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:37.377109 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" event={"ID":"b0ee52bc-2034-4487-8030-93c61a828509","Type":"ContainerStarted","Data":"097461543cfb31a8691986cfb07594d90c245552b59077a1e6a29249564bc546"} Sep 11 14:38:38.156512 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:38.156467 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:38.156869 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:38.156839 2553 patch_prober.go:28] interesting pod/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:38:38.156963 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:38.156902 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" podUID="b0ee52bc-2034-4487-8030-93c61a828509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 14:38:38.663653 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:38.663571 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z" podStartSLOduration=14.663550197 podStartE2EDuration="14.663550197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:38:26.357079441 +0000 UTC m=+1416.971185785" watchObservedRunningTime="2026-09-11 14:38:38.663550197 +0000 UTC m=+1429.277656542" Sep 11 14:38:39.156567 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:39.156526 2553 patch_prober.go:28] interesting pod/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:38:39.156776 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:39.156612 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" podUID="b0ee52bc-2034-4487-8030-93c61a828509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 14:38:40.156539 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:40.156506 2553 patch_prober.go:28] interesting pod/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:38:40.157148 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:40.157095 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" podUID="b0ee52bc-2034-4487-8030-93c61a828509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 14:38:41.159896 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:41.159851 2553 patch_prober.go:28] interesting pod/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:38:41.160528 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:41.160493 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" podUID="b0ee52bc-2034-4487-8030-93c61a828509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 14:38:42.157093 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:42.156983 2553 patch_prober.go:28] interesting pod/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:38:42.157093 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:42.157060 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" podUID="b0ee52bc-2034-4487-8030-93c61a828509" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Sep 11 14:38:43.160230 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:43.160202 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:43.160686 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:43.160525 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:43.161203 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:43.161185 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:38:43.618978 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:43.618888 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:43.732084 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:43.732046 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:44.722771 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:44.722733 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:45.217570 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:45.217538 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:45.433868 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:45.433839 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:47.432427 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:47.432368 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:48.273529 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:48.273487 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:48.638862 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:48.638827 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:48.750972 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:48.750937 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:48.869923 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:48.869886 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:49.232095 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:49.232055 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:49.433651 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:49.433613 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:50.822796 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:50.822758 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:51.011682 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:51.011651 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:51.189894 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:51.189830 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:51.973653 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:51.973617 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:52.220483 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:52.220449 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:52.425998 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:52.425961 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4"] Sep 11 14:38:52.430846 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:52.430825 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4" Sep 11 14:38:52.438591 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:52.438568 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-28858c-dockercfg-2x9sv\"" Sep 11 14:38:52.440391 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:52.440371 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-28858c\"" Sep 11 14:38:52.455423 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:52.455381 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4"] Sep 11 14:38:52.526929 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:52.526891 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c13805ad-69a7-421e-aea0-89b18b767a2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4\" (UID: \"c13805ad-69a7-421e-aea0-89b18b767a2d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4" Sep 11 14:38:52.527107 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:52.527042 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8ppn\" (UniqueName: \"kubernetes.io/projected/c13805ad-69a7-421e-aea0-89b18b767a2d-kube-api-access-d8ppn\") pod \"payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4\" (UID: \"c13805ad-69a7-421e-aea0-89b18b767a2d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4" Sep 11 14:38:52.627812 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:52.627773 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c13805ad-69a7-421e-aea0-89b18b767a2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4\" (UID: \"c13805ad-69a7-421e-aea0-89b18b767a2d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4" Sep 11 14:38:52.627984 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:52.627866 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8ppn\" (UniqueName: \"kubernetes.io/projected/c13805ad-69a7-421e-aea0-89b18b767a2d-kube-api-access-d8ppn\") pod \"payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4\" (UID: \"c13805ad-69a7-421e-aea0-89b18b767a2d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4" Sep 11 14:38:52.628703 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:52.628682 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c13805ad-69a7-421e-aea0-89b18b767a2d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4\" (UID: \"c13805ad-69a7-421e-aea0-89b18b767a2d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4" Sep 11 14:38:52.642434 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:52.642383 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8ppn\" (UniqueName: \"kubernetes.io/projected/c13805ad-69a7-421e-aea0-89b18b767a2d-kube-api-access-d8ppn\") pod \"payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4\" (UID: \"c13805ad-69a7-421e-aea0-89b18b767a2d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4" Sep 11 14:38:52.743601 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:52.743515 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4" Sep 11 14:38:52.942486 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:52.942454 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:53.112880 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:53.112792 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:53.122379 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:53.122352 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4"] Sep 11 14:38:53.125553 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:38:53.125530 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13805ad_69a7_421e_aea0_89b18b767a2d.slice/crio-a6efb062a4337cf49615698be99487f67f18f3238393122230f3128865c5ea62 WatchSource:0}: Error finding container a6efb062a4337cf49615698be99487f67f18f3238393122230f3128865c5ea62: Status 404 returned error can't find the container with id a6efb062a4337cf49615698be99487f67f18f3238393122230f3128865c5ea62 Sep 11 14:38:53.247546 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:53.247507 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:53.460669 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:53.460629 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4" event={"ID":"c13805ad-69a7-421e-aea0-89b18b767a2d","Type":"ContainerStarted","Data":"33ba01b34714006173138e5e38c6a04d5150674a2c2099255a041536ddb51b12"} Sep 11 14:38:53.460827 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:53.460702 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4" event={"ID":"c13805ad-69a7-421e-aea0-89b18b767a2d","Type":"ContainerStarted","Data":"a6efb062a4337cf49615698be99487f67f18f3238393122230f3128865c5ea62"} Sep 11 14:38:53.460876 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:53.460857 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4" Sep 11 14:38:53.519112 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:53.519055 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4" podStartSLOduration=1.5190404100000001 podStartE2EDuration="1.51904041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:38:53.51684252 +0000 UTC m=+1444.130948864" watchObservedRunningTime="2026-09-11 14:38:53.51904041 +0000 UTC m=+1444.133146756" Sep 11 14:38:54.817783 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:54.817751 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:54.925433 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:54.925382 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:55.182181 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:55.182145 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:55.788233 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:55.788200 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:55.894777 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:55.894742 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:56.147201 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:56.147166 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4"] Sep 11 14:38:56.150717 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:56.150699 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4" Sep 11 14:38:56.155718 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:56.155699 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-977c12-dockercfg-tz8f8\"" Sep 11 14:38:56.156319 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:56.156296 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-977c12\"" Sep 11 14:38:56.196164 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:56.196136 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4"] Sep 11 14:38:56.262910 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:56.262869 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m7b6\" (UniqueName: \"kubernetes.io/projected/4fd81613-5178-4e10-8057-6cd282a6c4b7-kube-api-access-4m7b6\") pod \"payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4\" (UID: \"4fd81613-5178-4e10-8057-6cd282a6c4b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4" Sep 11 14:38:56.263090 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:56.262917 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fd81613-5178-4e10-8057-6cd282a6c4b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4\" (UID: \"4fd81613-5178-4e10-8057-6cd282a6c4b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4" Sep 11 14:38:56.363928 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:56.363890 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m7b6\" (UniqueName: \"kubernetes.io/projected/4fd81613-5178-4e10-8057-6cd282a6c4b7-kube-api-access-4m7b6\") pod \"payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4\" (UID: \"4fd81613-5178-4e10-8057-6cd282a6c4b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4" Sep 11 14:38:56.364138 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:56.363935 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fd81613-5178-4e10-8057-6cd282a6c4b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4\" (UID: \"4fd81613-5178-4e10-8057-6cd282a6c4b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4" Sep 11 14:38:56.364704 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:56.364683 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fd81613-5178-4e10-8057-6cd282a6c4b7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4\" (UID: \"4fd81613-5178-4e10-8057-6cd282a6c4b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4" Sep 11 14:38:56.377307 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:56.377272 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m7b6\" (UniqueName: \"kubernetes.io/projected/4fd81613-5178-4e10-8057-6cd282a6c4b7-kube-api-access-4m7b6\") pod \"payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4\" (UID: \"4fd81613-5178-4e10-8057-6cd282a6c4b7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4" Sep 11 14:38:56.461366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:56.461273 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4" Sep 11 14:38:56.749545 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:56.749509 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4"] Sep 11 14:38:56.752310 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:38:56.752271 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd81613_5178_4e10_8057_6cd282a6c4b7.slice/crio-884c997b662e312d9e31e92c110e431594497c3826c31975e07833c5d4789b80 WatchSource:0}: Error finding container 884c997b662e312d9e31e92c110e431594497c3826c31975e07833c5d4789b80: Status 404 returned error can't find the container with id 884c997b662e312d9e31e92c110e431594497c3826c31975e07833c5d4789b80 Sep 11 14:38:56.950850 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:56.950807 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:57.489766 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:57.489730 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4" event={"ID":"4fd81613-5178-4e10-8057-6cd282a6c4b7","Type":"ContainerStarted","Data":"553390698f71674866d7a3075cd026749d0a3c30a91d3093b35b88ab0f7f2073"} Sep 11 14:38:57.489987 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:57.489829 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4" Sep 11 14:38:57.489987 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:57.489856 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4" event={"ID":"4fd81613-5178-4e10-8057-6cd282a6c4b7","Type":"ContainerStarted","Data":"884c997b662e312d9e31e92c110e431594497c3826c31975e07833c5d4789b80"} Sep 11 14:38:57.523304 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:57.523240 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4" podStartSLOduration=1.523220623 podStartE2EDuration="1.523220623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:38:57.521366142 +0000 UTC m=+1448.135472486" watchObservedRunningTime="2026-09-11 14:38:57.523220623 +0000 UTC m=+1448.137326968" Sep 11 14:38:58.231753 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:58.231714 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:38:59.361836 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:38:59.361792 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:00.512739 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:00.512705 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:04.467613 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:04.467581 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4" Sep 11 14:39:08.497365 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:08.497332 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4" Sep 11 14:39:12.930500 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:12.930464 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:15.457163 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:15.457117 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:16.755186 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:16.755152 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:18.020112 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:18.020071 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:18.932759 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:18.932723 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:19.089761 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.089726 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:19.315018 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.314925 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:19.731046 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.731015 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm"] Sep 11 14:39:19.734603 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.734581 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.739252 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.739228 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-7eeb45-openshift-default-dockercfg-brp98\"" Sep 11 14:39:19.765687 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.765657 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm"] Sep 11 14:39:19.769114 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.769080 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-istio-data\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.769246 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.769175 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-credential-socket\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.769246 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.769201 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032871f2-166d-4a19-a966-b76fabfff473-istiod-ca-cert\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.769246 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.769233 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032871f2-166d-4a19-a966-b76fabfff473-istio-token\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.769388 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.769261 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvrx6\" (UniqueName: \"kubernetes.io/projected/032871f2-166d-4a19-a966-b76fabfff473-kube-api-access-xvrx6\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.769388 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.769295 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-workload-socket\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.769388 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.769351 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032871f2-166d-4a19-a966-b76fabfff473-istio-podinfo\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.769507 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.769447 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-workload-certs\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.769507 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.769492 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-istio-envoy\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.780375 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.780350 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:19.870695 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.870657 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-istio-data\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.870891 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.870723 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-credential-socket\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.870891 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.870744 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032871f2-166d-4a19-a966-b76fabfff473-istiod-ca-cert\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.870891 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.870775 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032871f2-166d-4a19-a966-b76fabfff473-istio-token\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.870891 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.870805 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvrx6\" (UniqueName: \"kubernetes.io/projected/032871f2-166d-4a19-a966-b76fabfff473-kube-api-access-xvrx6\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.870891 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.870836 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-workload-socket\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.870891 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.870873 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032871f2-166d-4a19-a966-b76fabfff473-istio-podinfo\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.871255 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.871021 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-workload-certs\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.871255 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.871082 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-istio-envoy\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.871255 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.871174 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-credential-socket\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.871462 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.871399 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-workload-socket\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.871462 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.871438 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-workload-certs\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.871632 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.871609 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-istio-data\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.871771 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.871712 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032871f2-166d-4a19-a966-b76fabfff473-istiod-ca-cert\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.873890 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.873867 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-istio-envoy\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.874000 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.873872 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032871f2-166d-4a19-a966-b76fabfff473-istio-podinfo\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.883800 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.883777 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032871f2-166d-4a19-a966-b76fabfff473-istio-token\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.883912 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.883820 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvrx6\" (UniqueName: \"kubernetes.io/projected/032871f2-166d-4a19-a966-b76fabfff473-kube-api-access-xvrx6\") pod \"e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:19.953736 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:19.953694 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:20.045438 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:20.045330 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:20.199986 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:20.199951 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:20.347194 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:20.347160 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm"] Sep 11 14:39:20.347545 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:39:20.347522 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032871f2_166d_4a19_a966_b76fabfff473.slice/crio-be07a0cb64d29a67d922a5e1b7cd6e45c8f05b919834e83750592fe2cb678b54 WatchSource:0}: Error finding container be07a0cb64d29a67d922a5e1b7cd6e45c8f05b919834e83750592fe2cb678b54: Status 404 returned error can't find the container with id be07a0cb64d29a67d922a5e1b7cd6e45c8f05b919834e83750592fe2cb678b54 Sep 11 14:39:20.349986 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:20.349952 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:39:20.350065 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:20.350019 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:39:20.350065 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:20.350046 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:39:20.605652 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:20.605558 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" event={"ID":"032871f2-166d-4a19-a966-b76fabfff473","Type":"ContainerStarted","Data":"191ce6525130070439b47e91724682dfea46cf34eb34e9f42375d950d5a28890"} Sep 11 14:39:20.605652 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:20.605611 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" event={"ID":"032871f2-166d-4a19-a966-b76fabfff473","Type":"ContainerStarted","Data":"be07a0cb64d29a67d922a5e1b7cd6e45c8f05b919834e83750592fe2cb678b54"} Sep 11 14:39:20.649569 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:20.649510 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" podStartSLOduration=1.649493628 podStartE2EDuration="1.649493628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:39:20.646816025 +0000 UTC m=+1471.260922369" watchObservedRunningTime="2026-09-11 14:39:20.649493628 +0000 UTC m=+1471.263599988" Sep 11 14:39:21.046199 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:21.046169 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:21.051884 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:21.051857 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:21.611299 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:21.611266 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:21.612469 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:21.612451 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:21.832438 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:21.832378 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:22.816451 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:22.816395 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:22.975232 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:22.975193 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:23.101919 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:23.101829 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:23.657283 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:23.657232 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:25.299454 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:25.299398 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:27.146941 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:27.146906 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:27.252136 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:27.252106 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:27.823183 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:27.823145 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:28.134363 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:28.134259 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:28.270672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:28.270624 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:28.393798 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:28.393757 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:28.461345 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:28.461317 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q"] Sep 11 14:39:28.465053 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:28.465036 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q" Sep 11 14:39:28.479708 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:39:28.479685 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"payload-processing-plugins-e2e-shared-b-w2-7eeb45\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-7eeb45\"" type="*v1.ConfigMap" Sep 11 14:39:28.488052 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:39:28.488024 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="cdbd0164-2589-4690-8fa3-0aeb4d5a506a" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q" Sep 11 14:39:28.516764 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:28.516744 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q"] Sep 11 14:39:28.552877 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:28.552848 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbd0164-2589-4690-8fa3-0aeb4d5a506a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q\" (UID: \"cdbd0164-2589-4690-8fa3-0aeb4d5a506a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q" Sep 11 14:39:28.553013 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:28.552907 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tf8b\" (UniqueName: \"kubernetes.io/projected/cdbd0164-2589-4690-8fa3-0aeb4d5a506a-kube-api-access-4tf8b\") pod \"payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q\" (UID: \"cdbd0164-2589-4690-8fa3-0aeb4d5a506a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q" Sep 11 14:39:28.653592 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:28.653500 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbd0164-2589-4690-8fa3-0aeb4d5a506a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q\" (UID: \"cdbd0164-2589-4690-8fa3-0aeb4d5a506a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q" Sep 11 14:39:28.653756 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:28.653591 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tf8b\" (UniqueName: \"kubernetes.io/projected/cdbd0164-2589-4690-8fa3-0aeb4d5a506a-kube-api-access-4tf8b\") pod \"payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q\" (UID: \"cdbd0164-2589-4690-8fa3-0aeb4d5a506a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q" Sep 11 14:39:28.665721 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:28.665697 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tf8b\" (UniqueName: \"kubernetes.io/projected/cdbd0164-2589-4690-8fa3-0aeb4d5a506a-kube-api-access-4tf8b\") pod \"payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q\" (UID: \"cdbd0164-2589-4690-8fa3-0aeb4d5a506a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q" Sep 11 14:39:28.976849 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:28.976765 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:29.653985 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:39:29.653959 2553 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-shared-b-w2-7eeb45: failed to sync configmap cache: timed out waiting for the condition Sep 11 14:39:29.654380 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:39:29.654036 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cdbd0164-2589-4690-8fa3-0aeb4d5a506a-plugins-config-volume podName:cdbd0164-2589-4690-8fa3-0aeb4d5a506a nodeName:}" failed. No retries permitted until 2026-09-11 14:39:30.154019642 +0000 UTC m=+1480.768125964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cdbd0164-2589-4690-8fa3-0aeb4d5a506a-plugins-config-volume") pod "payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q" (UID: "cdbd0164-2589-4690-8fa3-0aeb4d5a506a") : failed to sync configmap cache: timed out waiting for the condition Sep 11 14:39:29.713697 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:29.713669 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-7eeb45\"" Sep 11 14:39:30.167355 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:30.167318 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbd0164-2589-4690-8fa3-0aeb4d5a506a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q\" (UID: \"cdbd0164-2589-4690-8fa3-0aeb4d5a506a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q" Sep 11 14:39:30.168000 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:30.167978 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbd0164-2589-4690-8fa3-0aeb4d5a506a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q\" (UID: \"cdbd0164-2589-4690-8fa3-0aeb4d5a506a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q" Sep 11 14:39:30.275129 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:30.275091 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q" Sep 11 14:39:30.308633 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:30.308600 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:30.564249 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:30.564224 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q"] Sep 11 14:39:30.565778 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:39:30.565745 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdbd0164_2589_4690_8fa3_0aeb4d5a506a.slice/crio-466e3f4a820fb0539d5117a03b086f8c6ddfac4f86b0b3d546221e104176e293 WatchSource:0}: Error finding container 466e3f4a820fb0539d5117a03b086f8c6ddfac4f86b0b3d546221e104176e293: Status 404 returned error can't find the container with id 466e3f4a820fb0539d5117a03b086f8c6ddfac4f86b0b3d546221e104176e293 Sep 11 14:39:30.662484 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:30.662449 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q" event={"ID":"cdbd0164-2589-4690-8fa3-0aeb4d5a506a","Type":"ContainerStarted","Data":"8a33034967fbe5dbf8c525a4e751239c577afc6997ae65039b22389bd69502f8"} Sep 11 14:39:30.662917 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:30.662509 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q" event={"ID":"cdbd0164-2589-4690-8fa3-0aeb4d5a506a","Type":"ContainerStarted","Data":"466e3f4a820fb0539d5117a03b086f8c6ddfac4f86b0b3d546221e104176e293"} Sep 11 14:39:30.662917 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:30.662563 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q" Sep 11 14:39:30.715803 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:30.715751 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q" podStartSLOduration=2.715732772 podStartE2EDuration="2.715732772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:39:30.713082691 +0000 UTC m=+1481.327189035" watchObservedRunningTime="2026-09-11 14:39:30.715732772 +0000 UTC m=+1481.329839115" Sep 11 14:39:31.614989 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:31.614949 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:41.671761 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:41.671725 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q" Sep 11 14:39:42.733148 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:42.733115 2553 generic.go:356] "Generic (PLEG): container finished" podID="032871f2-166d-4a19-a966-b76fabfff473" containerID="191ce6525130070439b47e91724682dfea46cf34eb34e9f42375d950d5a28890" exitCode=0 Sep 11 14:39:42.733641 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:42.733183 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" event={"ID":"032871f2-166d-4a19-a966-b76fabfff473","Type":"ContainerDied","Data":"191ce6525130070439b47e91724682dfea46cf34eb34e9f42375d950d5a28890"} Sep 11 14:39:42.733641 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:42.733603 2553 scope.go:117] "RemoveContainer" containerID="191ce6525130070439b47e91724682dfea46cf34eb34e9f42375d950d5a28890" Sep 11 14:39:42.734660 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:42.734629 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:39:42.734748 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:42.734696 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:39:42.734748 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:42.734728 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:39:43.740144 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:43.740107 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" event={"ID":"032871f2-166d-4a19-a966-b76fabfff473","Type":"ContainerStarted","Data":"7710cfd01c2169c3f6082a1d011b914081a7d6dd7c3126fc5360d9773371db30"} Sep 11 14:39:44.046654 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:44.046552 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:44.046901 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:44.046868 2553 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 14:39:44.046987 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:44.046940 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" podUID="032871f2-166d-4a19-a966-b76fabfff473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 14:39:45.046841 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:45.046801 2553 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 14:39:45.047838 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:45.047803 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" podUID="032871f2-166d-4a19-a966-b76fabfff473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 14:39:45.625732 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:45.625701 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:46.046365 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:46.046326 2553 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 14:39:46.046679 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:46.046650 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" podUID="032871f2-166d-4a19-a966-b76fabfff473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 14:39:47.046860 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:47.046824 2553 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 14:39:47.047679 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:47.047644 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" podUID="032871f2-166d-4a19-a966-b76fabfff473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 14:39:48.046794 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:48.046744 2553 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Sep 11 14:39:48.047123 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:48.047090 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" podUID="032871f2-166d-4a19-a966-b76fabfff473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Sep 11 14:39:49.050480 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:49.050450 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:49.050854 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:49.050832 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:49.051478 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:49.051454 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:39:49.956936 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:49.956909 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmqjh_6b2452d9-1730-458c-b380-dbcfd9e5be19/ovn-acl-logging/0.log" Sep 11 14:39:49.961952 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:49.961921 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmqjh_6b2452d9-1730-458c-b380-dbcfd9e5be19/ovn-acl-logging/0.log" Sep 11 14:39:51.129980 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:51.129949 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:52.499265 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:52.499229 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:52.941107 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:52.941073 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:53.482654 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:53.482615 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:53.788511 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:53.788434 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7863-028799"] Sep 11 14:39:53.792047 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:53.792032 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-028799" Sep 11 14:39:53.798969 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:53.798944 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vdvtq\"" Sep 11 14:39:53.799980 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:53.799939 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb7mf\" (UniqueName: \"kubernetes.io/projected/4d433c8e-be07-4a7c-94bf-3d0d32882b27-kube-api-access-rb7mf\") pod \"test-curl-7863-028799\" (UID: \"4d433c8e-be07-4a7c-94bf-3d0d32882b27\") " pod="opendatahub/test-curl-7863-028799" Sep 11 14:39:53.804159 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:53.804134 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7863-028799"] Sep 11 14:39:53.900318 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:53.900287 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb7mf\" (UniqueName: \"kubernetes.io/projected/4d433c8e-be07-4a7c-94bf-3d0d32882b27-kube-api-access-rb7mf\") pod \"test-curl-7863-028799\" (UID: \"4d433c8e-be07-4a7c-94bf-3d0d32882b27\") " pod="opendatahub/test-curl-7863-028799" Sep 11 14:39:53.915238 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:53.915213 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb7mf\" (UniqueName: \"kubernetes.io/projected/4d433c8e-be07-4a7c-94bf-3d0d32882b27-kube-api-access-rb7mf\") pod \"test-curl-7863-028799\" (UID: \"4d433c8e-be07-4a7c-94bf-3d0d32882b27\") " pod="opendatahub/test-curl-7863-028799" Sep 11 14:39:54.101262 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:54.101167 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-028799" Sep 11 14:39:54.388582 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:54.388554 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7863-028799"] Sep 11 14:39:54.391252 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:39:54.391228 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d433c8e_be07_4a7c_94bf_3d0d32882b27.slice/crio-6105369c3d188a9418c22456148691c5f57619abdc7a0fbf016e805b7d129c4b WatchSource:0}: Error finding container 6105369c3d188a9418c22456148691c5f57619abdc7a0fbf016e805b7d129c4b: Status 404 returned error can't find the container with id 6105369c3d188a9418c22456148691c5f57619abdc7a0fbf016e805b7d129c4b Sep 11 14:39:54.534094 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:54.534064 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:54.807407 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:54.807317 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7863-028799" event={"ID":"4d433c8e-be07-4a7c-94bf-3d0d32882b27","Type":"ContainerStarted","Data":"ef06165f8c9acc712eff3e7e76d8a059ec64edba70637e0bb7a4b13514b8ee4c"} Sep 11 14:39:54.807407 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:54.807363 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7863-028799" event={"ID":"4d433c8e-be07-4a7c-94bf-3d0d32882b27","Type":"ContainerStarted","Data":"6105369c3d188a9418c22456148691c5f57619abdc7a0fbf016e805b7d129c4b"} Sep 11 14:39:54.831349 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:54.831295 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7863-028799" podStartSLOduration=1.831278353 podStartE2EDuration="1.831278353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:39:54.830333212 +0000 UTC m=+1505.444439558" watchObservedRunningTime="2026-09-11 14:39:54.831278353 +0000 UTC m=+1505.445384696" Sep 11 14:39:55.741972 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:55.741940 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:39:55.828603 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:55.828519 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7863-028799"] Sep 11 14:39:55.832643 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:55.832615 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7863-028799"] Sep 11 14:39:55.974081 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:55.974047 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7863-9bb629"] Sep 11 14:39:55.974455 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:55.974442 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d433c8e-be07-4a7c-94bf-3d0d32882b27" containerName="test-curl-7863-028799" Sep 11 14:39:55.974507 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:55.974457 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d433c8e-be07-4a7c-94bf-3d0d32882b27" containerName="test-curl-7863-028799" Sep 11 14:39:55.974563 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:55.974553 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d433c8e-be07-4a7c-94bf-3d0d32882b27" containerName="test-curl-7863-028799" Sep 11 14:39:55.977333 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:55.977317 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-9bb629" Sep 11 14:39:56.000372 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:56.000346 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7863-9bb629"] Sep 11 14:39:56.017660 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:56.017628 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xw2w\" (UniqueName: \"kubernetes.io/projected/08f10f01-ef4a-4511-b2f8-086e43bef605-kube-api-access-7xw2w\") pod \"test-curl-7863-9bb629\" (UID: \"08f10f01-ef4a-4511-b2f8-086e43bef605\") " pod="opendatahub/test-curl-7863-9bb629" Sep 11 14:39:56.119017 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:56.118925 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xw2w\" (UniqueName: \"kubernetes.io/projected/08f10f01-ef4a-4511-b2f8-086e43bef605-kube-api-access-7xw2w\") pod \"test-curl-7863-9bb629\" (UID: \"08f10f01-ef4a-4511-b2f8-086e43bef605\") " pod="opendatahub/test-curl-7863-9bb629" Sep 11 14:39:56.131748 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:56.131721 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xw2w\" (UniqueName: \"kubernetes.io/projected/08f10f01-ef4a-4511-b2f8-086e43bef605-kube-api-access-7xw2w\") pod \"test-curl-7863-9bb629\" (UID: \"08f10f01-ef4a-4511-b2f8-086e43bef605\") " pod="opendatahub/test-curl-7863-9bb629" Sep 11 14:39:56.286296 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:56.286261 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-9bb629" Sep 11 14:39:56.569370 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:56.569347 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7863-9bb629"] Sep 11 14:39:56.570863 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:39:56.570837 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08f10f01_ef4a_4511_b2f8_086e43bef605.slice/crio-8243d8974d9ec8002c949bfd4da1df74f031b1853620636282476c199d8a8a23 WatchSource:0}: Error finding container 8243d8974d9ec8002c949bfd4da1df74f031b1853620636282476c199d8a8a23: Status 404 returned error can't find the container with id 8243d8974d9ec8002c949bfd4da1df74f031b1853620636282476c199d8a8a23 Sep 11 14:39:56.822260 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:56.822159 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7863-9bb629" event={"ID":"08f10f01-ef4a-4511-b2f8-086e43bef605","Type":"ContainerStarted","Data":"851f26cb15139afb38f762d2af65c7188ee8a34a5a5f503f6f7733ad788552d2"} Sep 11 14:39:56.822260 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:56.822207 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7863-9bb629" event={"ID":"08f10f01-ef4a-4511-b2f8-086e43bef605","Type":"ContainerStarted","Data":"8243d8974d9ec8002c949bfd4da1df74f031b1853620636282476c199d8a8a23"} Sep 11 14:39:56.822527 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:56.822231 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7863-028799" podUID="4d433c8e-be07-4a7c-94bf-3d0d32882b27" containerName="test-curl-7863-028799" containerID="cri-o://ef06165f8c9acc712eff3e7e76d8a059ec64edba70637e0bb7a4b13514b8ee4c" gracePeriod=2 Sep 11 14:39:56.844302 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:56.844255 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7863-9bb629" podStartSLOduration=1.844239132 podStartE2EDuration="1.844239132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:39:56.844153122 +0000 UTC m=+1507.458259466" watchObservedRunningTime="2026-09-11 14:39:56.844239132 +0000 UTC m=+1507.458345476" Sep 11 14:39:57.758459 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:57.758406 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7863-9bb629"] Sep 11 14:39:57.762683 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:57.762647 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7863-9bb629"] Sep 11 14:39:57.826984 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:39:57.826952 2553 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7863-9bb629" container="test-curl-7863-9bb629" Sep 11 14:39:57.917002 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:57.916970 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7863-ce8af4"] Sep 11 14:39:57.917383 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:57.917367 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08f10f01-ef4a-4511-b2f8-086e43bef605" containerName="test-curl-7863-9bb629" Sep 11 14:39:57.917448 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:57.917386 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="08f10f01-ef4a-4511-b2f8-086e43bef605" containerName="test-curl-7863-9bb629" Sep 11 14:39:57.917490 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:57.917484 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="08f10f01-ef4a-4511-b2f8-086e43bef605" containerName="test-curl-7863-9bb629" Sep 11 14:39:57.920397 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:57.920383 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-ce8af4" Sep 11 14:39:57.935649 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:57.935620 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8gzx\" (UniqueName: \"kubernetes.io/projected/90a0fae4-3d6b-4bcf-833f-cabd01e736e7-kube-api-access-l8gzx\") pod \"test-curl-7863-ce8af4\" (UID: \"90a0fae4-3d6b-4bcf-833f-cabd01e736e7\") " pod="opendatahub/test-curl-7863-ce8af4" Sep 11 14:39:57.935800 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:57.935663 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7863-ce8af4"] Sep 11 14:39:58.037058 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:58.036954 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8gzx\" (UniqueName: \"kubernetes.io/projected/90a0fae4-3d6b-4bcf-833f-cabd01e736e7-kube-api-access-l8gzx\") pod \"test-curl-7863-ce8af4\" (UID: \"90a0fae4-3d6b-4bcf-833f-cabd01e736e7\") " pod="opendatahub/test-curl-7863-ce8af4" Sep 11 14:39:58.048954 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:58.048914 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8gzx\" (UniqueName: \"kubernetes.io/projected/90a0fae4-3d6b-4bcf-833f-cabd01e736e7-kube-api-access-l8gzx\") pod \"test-curl-7863-ce8af4\" (UID: \"90a0fae4-3d6b-4bcf-833f-cabd01e736e7\") " pod="opendatahub/test-curl-7863-ce8af4" Sep 11 14:39:58.229554 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:58.229521 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-ce8af4" Sep 11 14:39:58.517510 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:58.517478 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7863-ce8af4"] Sep 11 14:39:58.520653 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:39:58.520623 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90a0fae4_3d6b_4bcf_833f_cabd01e736e7.slice/crio-6d449fe3490267ebc1f66634ff455049ce989ca86e009d48fd1f2488dbc39736 WatchSource:0}: Error finding container 6d449fe3490267ebc1f66634ff455049ce989ca86e009d48fd1f2488dbc39736: Status 404 returned error can't find the container with id 6d449fe3490267ebc1f66634ff455049ce989ca86e009d48fd1f2488dbc39736 Sep 11 14:39:58.834686 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:58.834658 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7863-ce8af4" event={"ID":"90a0fae4-3d6b-4bcf-833f-cabd01e736e7","Type":"ContainerStarted","Data":"a7bf39e4cb10aa78488818d14ebc27a4058c6b1f8ebc2207b6154201a0efa1e1"} Sep 11 14:39:58.834812 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:58.834697 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7863-ce8af4" event={"ID":"90a0fae4-3d6b-4bcf-833f-cabd01e736e7","Type":"ContainerStarted","Data":"6d449fe3490267ebc1f66634ff455049ce989ca86e009d48fd1f2488dbc39736"} Sep 11 14:39:58.834878 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:58.834792 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7863-9bb629" podUID="08f10f01-ef4a-4511-b2f8-086e43bef605" containerName="test-curl-7863-9bb629" containerID="cri-o://851f26cb15139afb38f762d2af65c7188ee8a34a5a5f503f6f7733ad788552d2" gracePeriod=2 Sep 11 14:39:58.858491 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:58.858444 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7863-ce8af4" podStartSLOduration=1.858403427 podStartE2EDuration="1.858403427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:39:58.857990716 +0000 UTC m=+1509.472097060" watchObservedRunningTime="2026-09-11 14:39:58.858403427 +0000 UTC m=+1509.472509770" Sep 11 14:39:58.991792 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:58.991757 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-028799" Sep 11 14:39:59.046131 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:59.046104 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb7mf\" (UniqueName: \"kubernetes.io/projected/4d433c8e-be07-4a7c-94bf-3d0d32882b27-kube-api-access-rb7mf\") pod \"4d433c8e-be07-4a7c-94bf-3d0d32882b27\" (UID: \"4d433c8e-be07-4a7c-94bf-3d0d32882b27\") " Sep 11 14:39:59.048086 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:59.048051 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d433c8e-be07-4a7c-94bf-3d0d32882b27-kube-api-access-rb7mf" (OuterVolumeSpecName: "kube-api-access-rb7mf") pod "4d433c8e-be07-4a7c-94bf-3d0d32882b27" (UID: "4d433c8e-be07-4a7c-94bf-3d0d32882b27"). InnerVolumeSpecName "kube-api-access-rb7mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:39:59.147286 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:59.147254 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb7mf\" (UniqueName: \"kubernetes.io/projected/4d433c8e-be07-4a7c-94bf-3d0d32882b27-kube-api-access-rb7mf\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:39:59.609867 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:59.609770 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7863-ce8af4"] Sep 11 14:39:59.617560 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:59.617531 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7863-ce8af4"] Sep 11 14:39:59.762570 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:59.762404 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7863-f5cd9d"] Sep 11 14:39:59.762874 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:59.762861 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90a0fae4-3d6b-4bcf-833f-cabd01e736e7" containerName="test-curl-7863-ce8af4" Sep 11 14:39:59.762912 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:59.762875 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="90a0fae4-3d6b-4bcf-833f-cabd01e736e7" containerName="test-curl-7863-ce8af4" Sep 11 14:39:59.762966 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:59.762939 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="90a0fae4-3d6b-4bcf-833f-cabd01e736e7" containerName="test-curl-7863-ce8af4" Sep 11 14:39:59.766127 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:59.766109 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-f5cd9d" Sep 11 14:39:59.778915 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:59.778888 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7863-f5cd9d"] Sep 11 14:39:59.841643 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:59.841606 2553 generic.go:356] "Generic (PLEG): container finished" podID="4d433c8e-be07-4a7c-94bf-3d0d32882b27" containerID="ef06165f8c9acc712eff3e7e76d8a059ec64edba70637e0bb7a4b13514b8ee4c" exitCode=137 Sep 11 14:39:59.841788 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:59.841665 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-028799" Sep 11 14:39:59.841788 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:59.841707 2553 scope.go:117] "RemoveContainer" containerID="ef06165f8c9acc712eff3e7e76d8a059ec64edba70637e0bb7a4b13514b8ee4c" Sep 11 14:39:59.842007 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:39:59.841990 2553 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7863-ce8af4" container="test-curl-7863-ce8af4" Sep 11 14:39:59.851916 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:59.851898 2553 scope.go:117] "RemoveContainer" containerID="ef06165f8c9acc712eff3e7e76d8a059ec64edba70637e0bb7a4b13514b8ee4c" Sep 11 14:39:59.852215 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:39:59.852196 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef06165f8c9acc712eff3e7e76d8a059ec64edba70637e0bb7a4b13514b8ee4c\": container with ID starting with ef06165f8c9acc712eff3e7e76d8a059ec64edba70637e0bb7a4b13514b8ee4c not found: ID does not exist" containerID="ef06165f8c9acc712eff3e7e76d8a059ec64edba70637e0bb7a4b13514b8ee4c" Sep 11 14:39:59.852277 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:59.852225 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef06165f8c9acc712eff3e7e76d8a059ec64edba70637e0bb7a4b13514b8ee4c"} err="failed to get container status \"ef06165f8c9acc712eff3e7e76d8a059ec64edba70637e0bb7a4b13514b8ee4c\": rpc error: code = NotFound desc = could not find container \"ef06165f8c9acc712eff3e7e76d8a059ec64edba70637e0bb7a4b13514b8ee4c\": container with ID starting with ef06165f8c9acc712eff3e7e76d8a059ec64edba70637e0bb7a4b13514b8ee4c not found: ID does not exist" Sep 11 14:39:59.852457 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:39:59.852434 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7863-028799\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="4d433c8e-be07-4a7c-94bf-3d0d32882b27" pod="opendatahub/test-curl-7863-028799" Sep 11 14:39:59.854872 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:59.854856 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ttzd\" (UniqueName: \"kubernetes.io/projected/8ceddd3f-a429-44f9-821e-bd19aaf596b9-kube-api-access-8ttzd\") pod \"test-curl-7863-f5cd9d\" (UID: \"8ceddd3f-a429-44f9-821e-bd19aaf596b9\") " pod="opendatahub/test-curl-7863-f5cd9d" Sep 11 14:39:59.912022 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:39:59.911995 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7863-028799\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="4d433c8e-be07-4a7c-94bf-3d0d32882b27" pod="opendatahub/test-curl-7863-028799" Sep 11 14:39:59.912627 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:59.912606 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d433c8e-be07-4a7c-94bf-3d0d32882b27" path="/var/lib/kubelet/pods/4d433c8e-be07-4a7c-94bf-3d0d32882b27/volumes" Sep 11 14:39:59.956193 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:59.956163 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ttzd\" (UniqueName: \"kubernetes.io/projected/8ceddd3f-a429-44f9-821e-bd19aaf596b9-kube-api-access-8ttzd\") pod \"test-curl-7863-f5cd9d\" (UID: \"8ceddd3f-a429-44f9-821e-bd19aaf596b9\") " pod="opendatahub/test-curl-7863-f5cd9d" Sep 11 14:39:59.966666 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:39:59.966639 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ttzd\" (UniqueName: \"kubernetes.io/projected/8ceddd3f-a429-44f9-821e-bd19aaf596b9-kube-api-access-8ttzd\") pod \"test-curl-7863-f5cd9d\" (UID: \"8ceddd3f-a429-44f9-821e-bd19aaf596b9\") " pod="opendatahub/test-curl-7863-f5cd9d" Sep 11 14:40:00.075307 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:00.075269 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-f5cd9d" Sep 11 14:40:00.360823 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:00.360798 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7863-f5cd9d"] Sep 11 14:40:00.362901 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:40:00.362864 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ceddd3f_a429_44f9_821e_bd19aaf596b9.slice/crio-2bddffd42180e1e337a8e7f3cab570abe8c688c6e737a34823101963f49c1557 WatchSource:0}: Error finding container 2bddffd42180e1e337a8e7f3cab570abe8c688c6e737a34823101963f49c1557: Status 404 returned error can't find the container with id 2bddffd42180e1e337a8e7f3cab570abe8c688c6e737a34823101963f49c1557 Sep 11 14:40:00.850081 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:00.850049 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7863-f5cd9d" event={"ID":"8ceddd3f-a429-44f9-821e-bd19aaf596b9","Type":"ContainerStarted","Data":"a3ffb5d8c2fef3ecedcef3599ac8df414636315699b7c440be07645f23274e99"} Sep 11 14:40:00.850203 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:00.850096 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7863-f5cd9d" event={"ID":"8ceddd3f-a429-44f9-821e-bd19aaf596b9","Type":"ContainerStarted","Data":"2bddffd42180e1e337a8e7f3cab570abe8c688c6e737a34823101963f49c1557"} Sep 11 14:40:00.850203 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:00.850106 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7863-ce8af4" podUID="90a0fae4-3d6b-4bcf-833f-cabd01e736e7" containerName="test-curl-7863-ce8af4" containerID="cri-o://a7bf39e4cb10aa78488818d14ebc27a4058c6b1f8ebc2207b6154201a0efa1e1" gracePeriod=2 Sep 11 14:40:01.011763 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:01.011731 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-9bb629" Sep 11 14:40:01.066104 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:01.066076 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xw2w\" (UniqueName: \"kubernetes.io/projected/08f10f01-ef4a-4511-b2f8-086e43bef605-kube-api-access-7xw2w\") pod \"08f10f01-ef4a-4511-b2f8-086e43bef605\" (UID: \"08f10f01-ef4a-4511-b2f8-086e43bef605\") " Sep 11 14:40:01.068077 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:01.068041 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f10f01-ef4a-4511-b2f8-086e43bef605-kube-api-access-7xw2w" (OuterVolumeSpecName: "kube-api-access-7xw2w") pod "08f10f01-ef4a-4511-b2f8-086e43bef605" (UID: "08f10f01-ef4a-4511-b2f8-086e43bef605"). InnerVolumeSpecName "kube-api-access-7xw2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:40:01.167397 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:01.167365 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xw2w\" (UniqueName: \"kubernetes.io/projected/08f10f01-ef4a-4511-b2f8-086e43bef605-kube-api-access-7xw2w\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:40:01.616239 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:01.616186 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7863-f5cd9d" podStartSLOduration=2.6161653400000002 podStartE2EDuration="2.61616534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:00.875677401 +0000 UTC m=+1511.489783749" watchObservedRunningTime="2026-09-11 14:40:01.61616534 +0000 UTC m=+1512.230271724" Sep 11 14:40:01.618002 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:01.617976 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7863-f5cd9d"] Sep 11 14:40:01.622960 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:01.622930 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7863-f5cd9d"] Sep 11 14:40:01.857478 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:01.857440 2553 generic.go:356] "Generic (PLEG): container finished" podID="08f10f01-ef4a-4511-b2f8-086e43bef605" containerID="851f26cb15139afb38f762d2af65c7188ee8a34a5a5f503f6f7733ad788552d2" exitCode=137 Sep 11 14:40:01.857478 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:01.857469 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-9bb629" Sep 11 14:40:01.857703 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:01.857516 2553 scope.go:117] "RemoveContainer" containerID="851f26cb15139afb38f762d2af65c7188ee8a34a5a5f503f6f7733ad788552d2" Sep 11 14:40:01.857895 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:40:01.857877 2553 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7863-f5cd9d" container="test-curl-7863-f5cd9d" Sep 11 14:40:01.866215 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:01.866197 2553 scope.go:117] "RemoveContainer" containerID="851f26cb15139afb38f762d2af65c7188ee8a34a5a5f503f6f7733ad788552d2" Sep 11 14:40:01.866528 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:40:01.866486 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"851f26cb15139afb38f762d2af65c7188ee8a34a5a5f503f6f7733ad788552d2\": container with ID starting with 851f26cb15139afb38f762d2af65c7188ee8a34a5a5f503f6f7733ad788552d2 not found: ID does not exist" containerID="851f26cb15139afb38f762d2af65c7188ee8a34a5a5f503f6f7733ad788552d2" Sep 11 14:40:01.866528 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:01.866514 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"851f26cb15139afb38f762d2af65c7188ee8a34a5a5f503f6f7733ad788552d2"} err="failed to get container status \"851f26cb15139afb38f762d2af65c7188ee8a34a5a5f503f6f7733ad788552d2\": rpc error: code = NotFound desc = could not find container \"851f26cb15139afb38f762d2af65c7188ee8a34a5a5f503f6f7733ad788552d2\": container with ID starting with 851f26cb15139afb38f762d2af65c7188ee8a34a5a5f503f6f7733ad788552d2 not found: ID does not exist" Sep 11 14:40:01.868311 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:40:01.868293 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7863-9bb629\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="08f10f01-ef4a-4511-b2f8-086e43bef605" pod="opendatahub/test-curl-7863-9bb629" Sep 11 14:40:01.913126 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:01.913094 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08f10f01-ef4a-4511-b2f8-086e43bef605" path="/var/lib/kubelet/pods/08f10f01-ef4a-4511-b2f8-086e43bef605/volumes" Sep 11 14:40:02.316592 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.316523 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:02.694653 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.694620 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq"] Sep 11 14:40:02.695134 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.695115 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ceddd3f-a429-44f9-821e-bd19aaf596b9" containerName="test-curl-7863-f5cd9d" Sep 11 14:40:02.695249 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.695135 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ceddd3f-a429-44f9-821e-bd19aaf596b9" containerName="test-curl-7863-f5cd9d" Sep 11 14:40:02.695306 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.695268 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ceddd3f-a429-44f9-821e-bd19aaf596b9" containerName="test-curl-7863-f5cd9d" Sep 11 14:40:02.699795 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.699776 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.704064 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.704037 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3893129d-openshift-default-dockercfg-5fwgt\"" Sep 11 14:40:02.730306 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.730269 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq"] Sep 11 14:40:02.752773 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.752739 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:02.783046 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.783013 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-token\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.783219 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.783082 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-workload-certs\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.783219 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.783114 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-workload-socket\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.783219 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.783141 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istiod-ca-cert\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.783219 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.783167 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-envoy\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.783219 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.783198 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-podinfo\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.783482 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.783264 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-data\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.783482 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.783302 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2wfc\" (UniqueName: \"kubernetes.io/projected/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-kube-api-access-h2wfc\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.783482 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.783357 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-credential-socket\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.864623 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.864595 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7863-f5cd9d" podUID="8ceddd3f-a429-44f9-821e-bd19aaf596b9" containerName="test-curl-7863-f5cd9d" containerID="cri-o://a3ffb5d8c2fef3ecedcef3599ac8df414636315699b7c440be07645f23274e99" gracePeriod=2 Sep 11 14:40:02.884028 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.883997 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-token\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.884196 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.884069 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-workload-certs\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.884196 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.884100 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-workload-socket\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.884196 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.884119 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istiod-ca-cert\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.884196 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.884152 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-envoy\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.884446 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.884339 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-podinfo\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.884446 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.884393 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-data\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.884568 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.884448 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2wfc\" (UniqueName: \"kubernetes.io/projected/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-kube-api-access-h2wfc\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.884568 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.884483 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-credential-socket\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.884676 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.884573 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-workload-certs\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.884676 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.884603 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-workload-socket\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.884781 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.884739 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:02.884960 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.884941 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-credential-socket\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.885051 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.885030 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-data\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.885130 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.885113 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istiod-ca-cert\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.886675 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.886656 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-envoy\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.886790 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.886773 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-podinfo\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.902514 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.902476 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2wfc\" (UniqueName: \"kubernetes.io/projected/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-kube-api-access-h2wfc\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:02.913593 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:02.913554 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-token\") pod \"e2e-mt-3893129d-openshift-default-569db84884-62pqq\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:03.010701 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:03.010670 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:03.050928 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:03.050893 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-ce8af4" Sep 11 14:40:03.087079 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:03.087039 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8gzx\" (UniqueName: \"kubernetes.io/projected/90a0fae4-3d6b-4bcf-833f-cabd01e736e7-kube-api-access-l8gzx\") pod \"90a0fae4-3d6b-4bcf-833f-cabd01e736e7\" (UID: \"90a0fae4-3d6b-4bcf-833f-cabd01e736e7\") " Sep 11 14:40:03.089767 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:03.089730 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a0fae4-3d6b-4bcf-833f-cabd01e736e7-kube-api-access-l8gzx" (OuterVolumeSpecName: "kube-api-access-l8gzx") pod "90a0fae4-3d6b-4bcf-833f-cabd01e736e7" (UID: "90a0fae4-3d6b-4bcf-833f-cabd01e736e7"). InnerVolumeSpecName "kube-api-access-l8gzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:40:03.189142 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:03.189108 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8gzx\" (UniqueName: \"kubernetes.io/projected/90a0fae4-3d6b-4bcf-833f-cabd01e736e7-kube-api-access-l8gzx\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:40:03.297564 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:03.297485 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq"] Sep 11 14:40:03.301112 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:40:03.301083 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7444dcd1_ba89_4dd6_a013_c2fb82f0400a.slice/crio-051c31f9b22b731ccf07022ac221d7d4f42c1a94684903b446a4439ffb30d376 WatchSource:0}: Error finding container 051c31f9b22b731ccf07022ac221d7d4f42c1a94684903b446a4439ffb30d376: Status 404 returned error can't find the container with id 051c31f9b22b731ccf07022ac221d7d4f42c1a94684903b446a4439ffb30d376 Sep 11 14:40:03.303391 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:03.303357 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:40:03.303509 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:03.303455 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:40:03.303509 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:03.303500 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:40:03.871404 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:03.871372 2553 generic.go:356] "Generic (PLEG): container finished" podID="90a0fae4-3d6b-4bcf-833f-cabd01e736e7" containerID="a7bf39e4cb10aa78488818d14ebc27a4058c6b1f8ebc2207b6154201a0efa1e1" exitCode=137 Sep 11 14:40:03.871825 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:03.871443 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-ce8af4" Sep 11 14:40:03.871825 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:03.871445 2553 scope.go:117] "RemoveContainer" containerID="a7bf39e4cb10aa78488818d14ebc27a4058c6b1f8ebc2207b6154201a0efa1e1" Sep 11 14:40:03.873336 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:03.873311 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" event={"ID":"7444dcd1-ba89-4dd6-a013-c2fb82f0400a","Type":"ContainerStarted","Data":"8bd7e8447f6b85d73b00d7f90ac4e548d476b48fbf6174d07d90974c0e6aa13f"} Sep 11 14:40:03.873462 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:03.873370 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" event={"ID":"7444dcd1-ba89-4dd6-a013-c2fb82f0400a","Type":"ContainerStarted","Data":"051c31f9b22b731ccf07022ac221d7d4f42c1a94684903b446a4439ffb30d376"} Sep 11 14:40:03.881949 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:03.881933 2553 scope.go:117] "RemoveContainer" containerID="a7bf39e4cb10aa78488818d14ebc27a4058c6b1f8ebc2207b6154201a0efa1e1" Sep 11 14:40:03.882214 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:40:03.882198 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7bf39e4cb10aa78488818d14ebc27a4058c6b1f8ebc2207b6154201a0efa1e1\": container with ID starting with a7bf39e4cb10aa78488818d14ebc27a4058c6b1f8ebc2207b6154201a0efa1e1 not found: ID does not exist" containerID="a7bf39e4cb10aa78488818d14ebc27a4058c6b1f8ebc2207b6154201a0efa1e1" Sep 11 14:40:03.882261 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:03.882223 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7bf39e4cb10aa78488818d14ebc27a4058c6b1f8ebc2207b6154201a0efa1e1"} err="failed to get container status \"a7bf39e4cb10aa78488818d14ebc27a4058c6b1f8ebc2207b6154201a0efa1e1\": rpc error: code = NotFound desc = could not find container \"a7bf39e4cb10aa78488818d14ebc27a4058c6b1f8ebc2207b6154201a0efa1e1\": container with ID starting with a7bf39e4cb10aa78488818d14ebc27a4058c6b1f8ebc2207b6154201a0efa1e1 not found: ID does not exist" Sep 11 14:40:03.912423 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:03.912365 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" podStartSLOduration=1.91234955 podStartE2EDuration="1.91234955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:03.909917929 +0000 UTC m=+1514.524024272" watchObservedRunningTime="2026-09-11 14:40:03.91234955 +0000 UTC m=+1514.526455894" Sep 11 14:40:03.913037 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:03.913016 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90a0fae4-3d6b-4bcf-833f-cabd01e736e7" path="/var/lib/kubelet/pods/90a0fae4-3d6b-4bcf-833f-cabd01e736e7/volumes" Sep 11 14:40:03.913955 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:40:03.913932 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7863-ce8af4\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="90a0fae4-3d6b-4bcf-833f-cabd01e736e7" pod="opendatahub/test-curl-7863-ce8af4" Sep 11 14:40:04.011432 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:04.011393 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:04.017360 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:04.017335 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:04.880319 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:04.880290 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:04.881365 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:04.881279 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:05.050860 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:05.050833 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-f5cd9d" Sep 11 14:40:05.107730 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:05.107690 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ttzd\" (UniqueName: \"kubernetes.io/projected/8ceddd3f-a429-44f9-821e-bd19aaf596b9-kube-api-access-8ttzd\") pod \"8ceddd3f-a429-44f9-821e-bd19aaf596b9\" (UID: \"8ceddd3f-a429-44f9-821e-bd19aaf596b9\") " Sep 11 14:40:05.109768 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:05.109736 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ceddd3f-a429-44f9-821e-bd19aaf596b9-kube-api-access-8ttzd" (OuterVolumeSpecName: "kube-api-access-8ttzd") pod "8ceddd3f-a429-44f9-821e-bd19aaf596b9" (UID: "8ceddd3f-a429-44f9-821e-bd19aaf596b9"). InnerVolumeSpecName "kube-api-access-8ttzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:40:05.209373 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:05.209340 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ttzd\" (UniqueName: \"kubernetes.io/projected/8ceddd3f-a429-44f9-821e-bd19aaf596b9-kube-api-access-8ttzd\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:40:05.473928 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:05.473832 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:05.886743 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:05.886654 2553 generic.go:356] "Generic (PLEG): container finished" podID="8ceddd3f-a429-44f9-821e-bd19aaf596b9" containerID="a3ffb5d8c2fef3ecedcef3599ac8df414636315699b7c440be07645f23274e99" exitCode=137 Sep 11 14:40:05.886743 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:05.886691 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7863-f5cd9d" Sep 11 14:40:05.887239 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:05.886757 2553 scope.go:117] "RemoveContainer" containerID="a3ffb5d8c2fef3ecedcef3599ac8df414636315699b7c440be07645f23274e99" Sep 11 14:40:05.895992 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:05.895970 2553 scope.go:117] "RemoveContainer" containerID="a3ffb5d8c2fef3ecedcef3599ac8df414636315699b7c440be07645f23274e99" Sep 11 14:40:05.896320 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:40:05.896300 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3ffb5d8c2fef3ecedcef3599ac8df414636315699b7c440be07645f23274e99\": container with ID starting with a3ffb5d8c2fef3ecedcef3599ac8df414636315699b7c440be07645f23274e99 not found: ID does not exist" containerID="a3ffb5d8c2fef3ecedcef3599ac8df414636315699b7c440be07645f23274e99" Sep 11 14:40:05.896438 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:05.896327 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3ffb5d8c2fef3ecedcef3599ac8df414636315699b7c440be07645f23274e99"} err="failed to get container status \"a3ffb5d8c2fef3ecedcef3599ac8df414636315699b7c440be07645f23274e99\": rpc error: code = NotFound desc = could not find container \"a3ffb5d8c2fef3ecedcef3599ac8df414636315699b7c440be07645f23274e99\": container with ID starting with a3ffb5d8c2fef3ecedcef3599ac8df414636315699b7c440be07645f23274e99 not found: ID does not exist" Sep 11 14:40:05.901502 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:40:05.901402 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7863-f5cd9d\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="8ceddd3f-a429-44f9-821e-bd19aaf596b9" pod="opendatahub/test-curl-7863-f5cd9d" Sep 11 14:40:05.911628 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:05.911603 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ceddd3f-a429-44f9-821e-bd19aaf596b9" path="/var/lib/kubelet/pods/8ceddd3f-a429-44f9-821e-bd19aaf596b9/volumes" Sep 11 14:40:06.092805 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:06.092765 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:06.306716 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:06.306680 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:07.261131 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:07.261096 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:08.783650 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:08.783612 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:09.450329 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:09.450296 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:09.581601 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:09.581566 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:10.385534 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:10.385499 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:11.777321 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:11.777287 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:13.139267 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:13.139223 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:13.256080 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:13.256036 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:13.402648 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:13.402537 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:14.112482 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:14.112445 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:14.267353 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:14.267310 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:14.444391 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:14.444354 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv"] Sep 11 14:40:14.449526 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:14.449501 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv" Sep 11 14:40:14.467743 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:14.467721 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3893129d\"" Sep 11 14:40:14.487881 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:14.487859 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv"] Sep 11 14:40:14.602654 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:14.602624 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hplpm\" (UniqueName: \"kubernetes.io/projected/ebb90c86-b738-46f3-a0f1-dbd76af7b40c-kube-api-access-hplpm\") pod \"payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv\" (UID: \"ebb90c86-b738-46f3-a0f1-dbd76af7b40c\") " pod="openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv" Sep 11 14:40:14.602834 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:14.602671 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebb90c86-b738-46f3-a0f1-dbd76af7b40c-plugins-config-volume\") pod \"payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv\" (UID: \"ebb90c86-b738-46f3-a0f1-dbd76af7b40c\") " pod="openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv" Sep 11 14:40:14.703833 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:14.703747 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hplpm\" (UniqueName: \"kubernetes.io/projected/ebb90c86-b738-46f3-a0f1-dbd76af7b40c-kube-api-access-hplpm\") pod \"payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv\" (UID: \"ebb90c86-b738-46f3-a0f1-dbd76af7b40c\") " pod="openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv" Sep 11 14:40:14.703833 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:14.703784 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebb90c86-b738-46f3-a0f1-dbd76af7b40c-plugins-config-volume\") pod \"payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv\" (UID: \"ebb90c86-b738-46f3-a0f1-dbd76af7b40c\") " pod="openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv" Sep 11 14:40:14.704479 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:14.704460 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebb90c86-b738-46f3-a0f1-dbd76af7b40c-plugins-config-volume\") pod \"payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv\" (UID: \"ebb90c86-b738-46f3-a0f1-dbd76af7b40c\") " pod="openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv" Sep 11 14:40:14.721031 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:14.720996 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hplpm\" (UniqueName: \"kubernetes.io/projected/ebb90c86-b738-46f3-a0f1-dbd76af7b40c-kube-api-access-hplpm\") pod \"payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv\" (UID: \"ebb90c86-b738-46f3-a0f1-dbd76af7b40c\") " pod="openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv" Sep 11 14:40:14.759271 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:14.759234 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv" Sep 11 14:40:15.061192 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:15.061167 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv"] Sep 11 14:40:15.063536 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:40:15.063510 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb90c86_b738_46f3_a0f1_dbd76af7b40c.slice/crio-b6e529dc1f8881f8f7d2cffb860757b49c191caaf71c3b46ae8c3fd3966e6fb2 WatchSource:0}: Error finding container b6e529dc1f8881f8f7d2cffb860757b49c191caaf71c3b46ae8c3fd3966e6fb2: Status 404 returned error can't find the container with id b6e529dc1f8881f8f7d2cffb860757b49c191caaf71c3b46ae8c3fd3966e6fb2 Sep 11 14:40:15.942704 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:15.942672 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv" event={"ID":"ebb90c86-b738-46f3-a0f1-dbd76af7b40c","Type":"ContainerStarted","Data":"51184fcfaba0be052d6434ce09d3c0194a68b82f2e7ecfae8c849e501116bd0f"} Sep 11 14:40:15.943130 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:15.942726 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv" event={"ID":"ebb90c86-b738-46f3-a0f1-dbd76af7b40c","Type":"ContainerStarted","Data":"b6e529dc1f8881f8f7d2cffb860757b49c191caaf71c3b46ae8c3fd3966e6fb2"} Sep 11 14:40:15.943130 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:15.942779 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv" Sep 11 14:40:15.986300 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:15.986244 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv" podStartSLOduration=1.986222625 podStartE2EDuration="1.986222625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:15.985478229 +0000 UTC m=+1526.599584573" watchObservedRunningTime="2026-09-11 14:40:15.986222625 +0000 UTC m=+1526.600329019" Sep 11 14:40:26.951667 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:26.951623 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv" Sep 11 14:40:31.024851 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:31.024818 2553 generic.go:356] "Generic (PLEG): container finished" podID="7444dcd1-ba89-4dd6-a013-c2fb82f0400a" containerID="8bd7e8447f6b85d73b00d7f90ac4e548d476b48fbf6174d07d90974c0e6aa13f" exitCode=0 Sep 11 14:40:31.025237 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:31.024894 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" event={"ID":"7444dcd1-ba89-4dd6-a013-c2fb82f0400a","Type":"ContainerDied","Data":"8bd7e8447f6b85d73b00d7f90ac4e548d476b48fbf6174d07d90974c0e6aa13f"} Sep 11 14:40:31.025332 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:31.025318 2553 scope.go:117] "RemoveContainer" containerID="8bd7e8447f6b85d73b00d7f90ac4e548d476b48fbf6174d07d90974c0e6aa13f" Sep 11 14:40:31.026558 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:31.026526 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:40:31.026628 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:31.026600 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:40:31.026665 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:31.026633 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:40:32.031757 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:32.031725 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" event={"ID":"7444dcd1-ba89-4dd6-a013-c2fb82f0400a","Type":"ContainerStarted","Data":"8fae0b6ad3676bddbd35c686d04477c2f94134aa6a065cf0a70b444cad082114"} Sep 11 14:40:33.011389 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:33.011348 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:33.012279 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:33.012253 2553 patch_prober.go:28] interesting pod/e2e-mt-3893129d-openshift-default-569db84884-62pqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Sep 11 14:40:33.012408 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:33.012301 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" podUID="7444dcd1-ba89-4dd6-a013-c2fb82f0400a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Sep 11 14:40:34.011803 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:34.011767 2553 patch_prober.go:28] interesting pod/e2e-mt-3893129d-openshift-default-569db84884-62pqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Sep 11 14:40:34.012229 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:34.011822 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" podUID="7444dcd1-ba89-4dd6-a013-c2fb82f0400a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Sep 11 14:40:34.881238 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:34.881194 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:35.011484 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:35.011449 2553 patch_prober.go:28] interesting pod/e2e-mt-3893129d-openshift-default-569db84884-62pqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Sep 11 14:40:35.011667 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:35.011503 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" podUID="7444dcd1-ba89-4dd6-a013-c2fb82f0400a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Sep 11 14:40:36.011132 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:36.011097 2553 patch_prober.go:28] interesting pod/e2e-mt-3893129d-openshift-default-569db84884-62pqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Sep 11 14:40:36.011591 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:36.011158 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" podUID="7444dcd1-ba89-4dd6-a013-c2fb82f0400a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Sep 11 14:40:37.011290 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:37.011250 2553 patch_prober.go:28] interesting pod/e2e-mt-3893129d-openshift-default-569db84884-62pqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Sep 11 14:40:37.011733 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:37.011303 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" podUID="7444dcd1-ba89-4dd6-a013-c2fb82f0400a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Sep 11 14:40:38.015366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:38.015336 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:38.016294 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:38.016275 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:40:38.509767 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:38.509735 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:38.655693 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:38.655659 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:39.642669 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:39.642630 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:40.774301 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:40.774249 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:41.254009 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:41.253969 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:41.934584 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:41.934544 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:42.325585 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:42.325488 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:43.792961 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:43.792929 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:47.226430 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.226390 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946"] Sep 11 14:40:47.231223 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.231204 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:47.240589 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.240570 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3893129d\"/\"default-dockercfg-sfp6q\"" Sep 11 14:40:47.240669 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.240570 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3893129d\"/\"e2e-lifecycle-model-3893129d-kserve-self-signed-certs\"" Sep 11 14:40:47.240669 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.240622 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3893129d\"/\"kube-root-ca.crt\"" Sep 11 14:40:47.240669 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.240573 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3893129d\"/\"openshift-service-ca.crt\"" Sep 11 14:40:47.284938 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.284907 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946"] Sep 11 14:40:47.324393 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.324351 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpkc6\" (UniqueName: \"kubernetes.io/projected/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-kube-api-access-lpkc6\") pod \"e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:47.324567 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.324409 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-model-cache\") pod \"e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:47.324567 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.324501 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-dshm\") pod \"e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:47.324567 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.324535 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-home\") pod \"e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:47.324739 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.324636 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-tmp-dir\") pod \"e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:47.324739 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.324688 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-tls-certs\") pod \"e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:47.367724 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.367680 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:47.425325 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.425289 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-tls-certs\") pod \"e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:47.425548 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.425370 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpkc6\" (UniqueName: \"kubernetes.io/projected/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-kube-api-access-lpkc6\") pod \"e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:47.425548 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.425404 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-model-cache\") pod \"e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:47.425548 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.425476 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-dshm\") pod \"e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:47.425548 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.425506 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-home\") pod \"e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:47.425774 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.425567 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-tmp-dir\") pod \"e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:47.425892 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.425870 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-model-cache\") pod \"e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:47.426144 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.426123 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-home\") pod \"e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:47.426144 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.426139 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-tmp-dir\") pod \"e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:47.427966 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.427936 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-dshm\") pod \"e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:47.428570 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.428552 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-tls-certs\") pod \"e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:47.474476 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.474432 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpkc6\" (UniqueName: \"kubernetes.io/projected/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-kube-api-access-lpkc6\") pod \"e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:47.539844 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.539759 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:47.826835 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:47.826811 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946"] Sep 11 14:40:47.829093 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:40:47.829067 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8963620e_a60e_4e0c_8fc1_85e9094c8b7f.slice/crio-d2e44944aa992a44d9890cc4f7134df8f3ea6bb7515c758cc6ed4f98de5d708b WatchSource:0}: Error finding container d2e44944aa992a44d9890cc4f7134df8f3ea6bb7515c758cc6ed4f98de5d708b: Status 404 returned error can't find the container with id d2e44944aa992a44d9890cc4f7134df8f3ea6bb7515c758cc6ed4f98de5d708b Sep 11 14:40:48.132543 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:48.132450 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" event={"ID":"8963620e-a60e-4e0c-8fc1-85e9094c8b7f","Type":"ContainerStarted","Data":"c506ed0b7c5dfa4d511bc97527c53646e9eecc66ade9e7daec1ace80fc7a38a3"} Sep 11 14:40:48.132543 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:48.132502 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" event={"ID":"8963620e-a60e-4e0c-8fc1-85e9094c8b7f","Type":"ContainerStarted","Data":"d2e44944aa992a44d9890cc4f7134df8f3ea6bb7515c758cc6ed4f98de5d708b"} Sep 11 14:40:48.163606 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:48.163552 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" podStartSLOduration=1.1635335900000001 podStartE2EDuration="1.16353359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:40:48.162228392 +0000 UTC m=+1558.776334735" watchObservedRunningTime="2026-09-11 14:40:48.16353359 +0000 UTC m=+1558.777639935" Sep 11 14:40:54.762046 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:54.761999 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:40:57.540674 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:57.540636 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:57.541102 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:57.540691 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:57.553329 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:57.553302 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:40:58.198791 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:40:58.198762 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:41:07.973534 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:07.973496 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:08.056532 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:08.056500 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:21.567587 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:21.567543 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:22.823876 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:22.823835 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:23.748349 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:23.748315 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:25.003247 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:25.003199 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:26.047821 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:26.047781 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:26.890618 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:26.890580 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:28.139709 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:28.139676 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:37.223861 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:37.223817 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:37.292348 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:37.292310 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv"] Sep 11 14:41:37.292675 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:37.292651 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv" podUID="ebb90c86-b738-46f3-a0f1-dbd76af7b40c" containerName="payload-processing" containerID="cri-o://51184fcfaba0be052d6434ce09d3c0194a68b82f2e7ecfae8c849e501116bd0f" gracePeriod=30 Sep 11 14:41:37.356020 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:37.355988 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:37.474628 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:37.474550 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:37.546579 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:37.546554 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv" Sep 11 14:41:37.574403 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:37.574371 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:37.620962 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:37.620933 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hplpm\" (UniqueName: \"kubernetes.io/projected/ebb90c86-b738-46f3-a0f1-dbd76af7b40c-kube-api-access-hplpm\") pod \"ebb90c86-b738-46f3-a0f1-dbd76af7b40c\" (UID: \"ebb90c86-b738-46f3-a0f1-dbd76af7b40c\") " Sep 11 14:41:37.620962 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:37.620965 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebb90c86-b738-46f3-a0f1-dbd76af7b40c-plugins-config-volume\") pod \"ebb90c86-b738-46f3-a0f1-dbd76af7b40c\" (UID: \"ebb90c86-b738-46f3-a0f1-dbd76af7b40c\") " Sep 11 14:41:37.621483 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:37.621453 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb90c86-b738-46f3-a0f1-dbd76af7b40c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ebb90c86-b738-46f3-a0f1-dbd76af7b40c" (UID: "ebb90c86-b738-46f3-a0f1-dbd76af7b40c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:41:37.623050 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:37.623026 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb90c86-b738-46f3-a0f1-dbd76af7b40c-kube-api-access-hplpm" (OuterVolumeSpecName: "kube-api-access-hplpm") pod "ebb90c86-b738-46f3-a0f1-dbd76af7b40c" (UID: "ebb90c86-b738-46f3-a0f1-dbd76af7b40c"). InnerVolumeSpecName "kube-api-access-hplpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:41:37.656663 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:37.656542 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:37.722372 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:37.722337 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hplpm\" (UniqueName: \"kubernetes.io/projected/ebb90c86-b738-46f3-a0f1-dbd76af7b40c-kube-api-access-hplpm\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:41:37.722372 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:37.722364 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebb90c86-b738-46f3-a0f1-dbd76af7b40c-plugins-config-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:41:38.093359 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:38.093322 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:38.409803 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:38.409769 2553 generic.go:356] "Generic (PLEG): container finished" podID="ebb90c86-b738-46f3-a0f1-dbd76af7b40c" containerID="51184fcfaba0be052d6434ce09d3c0194a68b82f2e7ecfae8c849e501116bd0f" exitCode=0 Sep 11 14:41:38.410236 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:38.409816 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv" Sep 11 14:41:38.410236 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:38.409854 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv" event={"ID":"ebb90c86-b738-46f3-a0f1-dbd76af7b40c","Type":"ContainerDied","Data":"51184fcfaba0be052d6434ce09d3c0194a68b82f2e7ecfae8c849e501116bd0f"} Sep 11 14:41:38.410236 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:38.409904 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv" event={"ID":"ebb90c86-b738-46f3-a0f1-dbd76af7b40c","Type":"ContainerDied","Data":"b6e529dc1f8881f8f7d2cffb860757b49c191caaf71c3b46ae8c3fd3966e6fb2"} Sep 11 14:41:38.410236 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:38.409924 2553 scope.go:117] "RemoveContainer" containerID="51184fcfaba0be052d6434ce09d3c0194a68b82f2e7ecfae8c849e501116bd0f" Sep 11 14:41:38.418886 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:38.418868 2553 scope.go:117] "RemoveContainer" containerID="51184fcfaba0be052d6434ce09d3c0194a68b82f2e7ecfae8c849e501116bd0f" Sep 11 14:41:38.419166 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:41:38.419146 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51184fcfaba0be052d6434ce09d3c0194a68b82f2e7ecfae8c849e501116bd0f\": container with ID starting with 51184fcfaba0be052d6434ce09d3c0194a68b82f2e7ecfae8c849e501116bd0f not found: ID does not exist" containerID="51184fcfaba0be052d6434ce09d3c0194a68b82f2e7ecfae8c849e501116bd0f" Sep 11 14:41:38.419258 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:38.419172 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51184fcfaba0be052d6434ce09d3c0194a68b82f2e7ecfae8c849e501116bd0f"} err="failed to get container status \"51184fcfaba0be052d6434ce09d3c0194a68b82f2e7ecfae8c849e501116bd0f\": rpc error: code = NotFound desc = could not find container \"51184fcfaba0be052d6434ce09d3c0194a68b82f2e7ecfae8c849e501116bd0f\": container with ID starting with 51184fcfaba0be052d6434ce09d3c0194a68b82f2e7ecfae8c849e501116bd0f not found: ID does not exist" Sep 11 14:41:38.448512 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:38.448486 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv"] Sep 11 14:41:38.462979 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:38.462958 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3893129d-6657f5b789-9f9pv"] Sep 11 14:41:39.033168 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:39.033132 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:39.914308 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:39.914280 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebb90c86-b738-46f3-a0f1-dbd76af7b40c" path="/var/lib/kubelet/pods/ebb90c86-b738-46f3-a0f1-dbd76af7b40c/volumes" Sep 11 14:41:40.200237 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:40.200139 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:41.062364 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:41.062332 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:41.206733 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:41.206696 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:42.243174 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:42.243071 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:46.038990 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:46.038957 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946"] Sep 11 14:41:46.129223 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:46.129189 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:41:50.110681 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:50.110644 2553 scope.go:117] "RemoveContainer" containerID="b46bed8f188f97e6e07301e68967690b7ece03e9a140177c576ccdc5c81ef070" Sep 11 14:41:58.622073 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:41:58.622033 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:01.058318 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.058279 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" podUID="8963620e-a60e-4e0c-8fc1-85e9094c8b7f" containerName="main" containerID="cri-o://c506ed0b7c5dfa4d511bc97527c53646e9eecc66ade9e7daec1ace80fc7a38a3" gracePeriod=45 Sep 11 14:42:01.310464 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.310391 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:42:01.355676 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.355645 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-dshm\") pod \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " Sep 11 14:42:01.355863 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.355699 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpkc6\" (UniqueName: \"kubernetes.io/projected/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-kube-api-access-lpkc6\") pod \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " Sep 11 14:42:01.355863 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.355726 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-model-cache\") pod \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " Sep 11 14:42:01.355863 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.355787 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-tmp-dir\") pod \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " Sep 11 14:42:01.355863 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.355829 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-home\") pod \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " Sep 11 14:42:01.355863 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.355853 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-tls-certs\") pod \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\" (UID: \"8963620e-a60e-4e0c-8fc1-85e9094c8b7f\") " Sep 11 14:42:01.356460 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.356306 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-home" (OuterVolumeSpecName: "home") pod "8963620e-a60e-4e0c-8fc1-85e9094c8b7f" (UID: "8963620e-a60e-4e0c-8fc1-85e9094c8b7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:42:01.356460 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.356397 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "8963620e-a60e-4e0c-8fc1-85e9094c8b7f" (UID: "8963620e-a60e-4e0c-8fc1-85e9094c8b7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:42:01.356460 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.356436 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8963620e-a60e-4e0c-8fc1-85e9094c8b7f" (UID: "8963620e-a60e-4e0c-8fc1-85e9094c8b7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:42:01.358259 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.358222 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-kube-api-access-lpkc6" (OuterVolumeSpecName: "kube-api-access-lpkc6") pod "8963620e-a60e-4e0c-8fc1-85e9094c8b7f" (UID: "8963620e-a60e-4e0c-8fc1-85e9094c8b7f"). InnerVolumeSpecName "kube-api-access-lpkc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:42:01.358653 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.358621 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-dshm" (OuterVolumeSpecName: "dshm") pod "8963620e-a60e-4e0c-8fc1-85e9094c8b7f" (UID: "8963620e-a60e-4e0c-8fc1-85e9094c8b7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:42:01.358948 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.358916 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8963620e-a60e-4e0c-8fc1-85e9094c8b7f" (UID: "8963620e-a60e-4e0c-8fc1-85e9094c8b7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:42:01.457431 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.457382 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-dshm\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:42:01.457431 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.457427 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpkc6\" (UniqueName: \"kubernetes.io/projected/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-kube-api-access-lpkc6\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:42:01.457431 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.457443 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-model-cache\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:42:01.457682 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.457455 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-tmp-dir\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:42:01.457682 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.457465 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-home\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:42:01.457682 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.457475 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8963620e-a60e-4e0c-8fc1-85e9094c8b7f-tls-certs\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:42:01.533919 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.533885 2553 generic.go:356] "Generic (PLEG): container finished" podID="8963620e-a60e-4e0c-8fc1-85e9094c8b7f" containerID="c506ed0b7c5dfa4d511bc97527c53646e9eecc66ade9e7daec1ace80fc7a38a3" exitCode=0 Sep 11 14:42:01.534096 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.533932 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" Sep 11 14:42:01.534096 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.533975 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" event={"ID":"8963620e-a60e-4e0c-8fc1-85e9094c8b7f","Type":"ContainerDied","Data":"c506ed0b7c5dfa4d511bc97527c53646e9eecc66ade9e7daec1ace80fc7a38a3"} Sep 11 14:42:01.534096 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.534024 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946" event={"ID":"8963620e-a60e-4e0c-8fc1-85e9094c8b7f","Type":"ContainerDied","Data":"d2e44944aa992a44d9890cc4f7134df8f3ea6bb7515c758cc6ed4f98de5d708b"} Sep 11 14:42:01.534096 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.534042 2553 scope.go:117] "RemoveContainer" containerID="c506ed0b7c5dfa4d511bc97527c53646e9eecc66ade9e7daec1ace80fc7a38a3" Sep 11 14:42:01.543180 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.543161 2553 scope.go:117] "RemoveContainer" containerID="c506ed0b7c5dfa4d511bc97527c53646e9eecc66ade9e7daec1ace80fc7a38a3" Sep 11 14:42:01.543494 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:42:01.543472 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c506ed0b7c5dfa4d511bc97527c53646e9eecc66ade9e7daec1ace80fc7a38a3\": container with ID starting with c506ed0b7c5dfa4d511bc97527c53646e9eecc66ade9e7daec1ace80fc7a38a3 not found: ID does not exist" containerID="c506ed0b7c5dfa4d511bc97527c53646e9eecc66ade9e7daec1ace80fc7a38a3" Sep 11 14:42:01.543555 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.543507 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c506ed0b7c5dfa4d511bc97527c53646e9eecc66ade9e7daec1ace80fc7a38a3"} err="failed to get container status \"c506ed0b7c5dfa4d511bc97527c53646e9eecc66ade9e7daec1ace80fc7a38a3\": rpc error: code = NotFound desc = could not find container \"c506ed0b7c5dfa4d511bc97527c53646e9eecc66ade9e7daec1ace80fc7a38a3\": container with ID starting with c506ed0b7c5dfa4d511bc97527c53646e9eecc66ade9e7daec1ace80fc7a38a3 not found: ID does not exist" Sep 11 14:42:01.569736 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.569660 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946"] Sep 11 14:42:01.581838 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.581814 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3893129d/e2e-lifecycle-model-3893129d-kserve-7db464db4c-dm946"] Sep 11 14:42:01.913207 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:01.913173 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8963620e-a60e-4e0c-8fc1-85e9094c8b7f" path="/var/lib/kubelet/pods/8963620e-a60e-4e0c-8fc1-85e9094c8b7f/volumes" Sep 11 14:42:06.908076 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:06.908026 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:10.947385 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:10.947342 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:11.072249 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:11.072218 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:14.475909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:14.475874 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7866-ba617a"] Sep 11 14:42:14.476299 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:14.476286 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8963620e-a60e-4e0c-8fc1-85e9094c8b7f" containerName="main" Sep 11 14:42:14.476345 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:14.476301 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8963620e-a60e-4e0c-8fc1-85e9094c8b7f" containerName="main" Sep 11 14:42:14.476345 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:14.476320 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb90c86-b738-46f3-a0f1-dbd76af7b40c" containerName="payload-processing" Sep 11 14:42:14.476345 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:14.476326 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb90c86-b738-46f3-a0f1-dbd76af7b40c" containerName="payload-processing" Sep 11 14:42:14.476473 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:14.476385 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8963620e-a60e-4e0c-8fc1-85e9094c8b7f" containerName="main" Sep 11 14:42:14.476473 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:14.476393 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb90c86-b738-46f3-a0f1-dbd76af7b40c" containerName="payload-processing" Sep 11 14:42:14.479536 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:14.479521 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7866-ba617a" Sep 11 14:42:14.482948 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:14.482927 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vdvtq\"" Sep 11 14:42:14.493543 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:14.493519 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7866-ba617a"] Sep 11 14:42:14.581874 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:14.581837 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6qvx\" (UniqueName: \"kubernetes.io/projected/49eca948-49ea-4661-b3be-c98b9aa4cc16-kube-api-access-k6qvx\") pod \"test-curl-7866-ba617a\" (UID: \"49eca948-49ea-4661-b3be-c98b9aa4cc16\") " pod="opendatahub/test-curl-7866-ba617a" Sep 11 14:42:14.682801 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:14.682762 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6qvx\" (UniqueName: \"kubernetes.io/projected/49eca948-49ea-4661-b3be-c98b9aa4cc16-kube-api-access-k6qvx\") pod \"test-curl-7866-ba617a\" (UID: \"49eca948-49ea-4661-b3be-c98b9aa4cc16\") " pod="opendatahub/test-curl-7866-ba617a" Sep 11 14:42:14.693336 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:14.693303 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6qvx\" (UniqueName: \"kubernetes.io/projected/49eca948-49ea-4661-b3be-c98b9aa4cc16-kube-api-access-k6qvx\") pod \"test-curl-7866-ba617a\" (UID: \"49eca948-49ea-4661-b3be-c98b9aa4cc16\") " pod="opendatahub/test-curl-7866-ba617a" Sep 11 14:42:14.789365 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:14.789276 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7866-ba617a" Sep 11 14:42:15.063731 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:15.063708 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7866-ba617a"] Sep 11 14:42:15.066407 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:42:15.066380 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49eca948_49ea_4661_b3be_c98b9aa4cc16.slice/crio-b9a0b21c1b17b4bfb0e889bc2c1afd53a8b5adba2ef9d2a47cf16f7386c4b76f WatchSource:0}: Error finding container b9a0b21c1b17b4bfb0e889bc2c1afd53a8b5adba2ef9d2a47cf16f7386c4b76f: Status 404 returned error can't find the container with id b9a0b21c1b17b4bfb0e889bc2c1afd53a8b5adba2ef9d2a47cf16f7386c4b76f Sep 11 14:42:15.612105 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:15.612060 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7866-ba617a" event={"ID":"49eca948-49ea-4661-b3be-c98b9aa4cc16","Type":"ContainerStarted","Data":"003caffe98d8d820c44300e9598a76a5d6760dc73e91cb3eb7b3d69063923060"} Sep 11 14:42:15.612520 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:15.612114 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7866-ba617a" event={"ID":"49eca948-49ea-4661-b3be-c98b9aa4cc16","Type":"ContainerStarted","Data":"b9a0b21c1b17b4bfb0e889bc2c1afd53a8b5adba2ef9d2a47cf16f7386c4b76f"} Sep 11 14:42:15.635607 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:15.635551 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7866-ba617a" podStartSLOduration=1.63553493 podStartE2EDuration="1.63553493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:42:15.6347791 +0000 UTC m=+1646.248885443" watchObservedRunningTime="2026-09-11 14:42:15.63553493 +0000 UTC m=+1646.249641274" Sep 11 14:42:16.338802 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:16.338767 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7866-ba617a"] Sep 11 14:42:16.344466 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:16.344438 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7866-ba617a"] Sep 11 14:42:16.617299 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:42:16.617274 2553 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7866-ba617a" container="test-curl-7866-ba617a" Sep 11 14:42:17.622564 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:17.622524 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7866-ba617a" podUID="49eca948-49ea-4661-b3be-c98b9aa4cc16" containerName="test-curl-7866-ba617a" containerID="cri-o://003caffe98d8d820c44300e9598a76a5d6760dc73e91cb3eb7b3d69063923060" gracePeriod=2 Sep 11 14:42:17.625588 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:42:17.625559 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7866-ba617a\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="49eca948-49ea-4661-b3be-c98b9aa4cc16" pod="opendatahub/test-curl-7866-ba617a" Sep 11 14:42:18.014244 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:18.014208 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:19.072569 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:19.072527 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:19.782577 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:19.782550 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7866-ba617a" Sep 11 14:42:19.931144 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:19.931119 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6qvx\" (UniqueName: \"kubernetes.io/projected/49eca948-49ea-4661-b3be-c98b9aa4cc16-kube-api-access-k6qvx\") pod \"49eca948-49ea-4661-b3be-c98b9aa4cc16\" (UID: \"49eca948-49ea-4661-b3be-c98b9aa4cc16\") " Sep 11 14:42:19.933173 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:19.933144 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49eca948-49ea-4661-b3be-c98b9aa4cc16-kube-api-access-k6qvx" (OuterVolumeSpecName: "kube-api-access-k6qvx") pod "49eca948-49ea-4661-b3be-c98b9aa4cc16" (UID: "49eca948-49ea-4661-b3be-c98b9aa4cc16"). InnerVolumeSpecName "kube-api-access-k6qvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:42:20.032349 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:20.032316 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6qvx\" (UniqueName: \"kubernetes.io/projected/49eca948-49ea-4661-b3be-c98b9aa4cc16-kube-api-access-k6qvx\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:42:20.642277 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:20.642223 2553 generic.go:356] "Generic (PLEG): container finished" podID="49eca948-49ea-4661-b3be-c98b9aa4cc16" containerID="003caffe98d8d820c44300e9598a76a5d6760dc73e91cb3eb7b3d69063923060" exitCode=137 Sep 11 14:42:20.642277 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:20.642272 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7866-ba617a" Sep 11 14:42:20.642853 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:20.642317 2553 scope.go:117] "RemoveContainer" containerID="003caffe98d8d820c44300e9598a76a5d6760dc73e91cb3eb7b3d69063923060" Sep 11 14:42:20.652468 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:20.652447 2553 scope.go:117] "RemoveContainer" containerID="003caffe98d8d820c44300e9598a76a5d6760dc73e91cb3eb7b3d69063923060" Sep 11 14:42:20.652772 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:42:20.652746 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"003caffe98d8d820c44300e9598a76a5d6760dc73e91cb3eb7b3d69063923060\": container with ID starting with 003caffe98d8d820c44300e9598a76a5d6760dc73e91cb3eb7b3d69063923060 not found: ID does not exist" containerID="003caffe98d8d820c44300e9598a76a5d6760dc73e91cb3eb7b3d69063923060" Sep 11 14:42:20.652860 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:20.652786 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"003caffe98d8d820c44300e9598a76a5d6760dc73e91cb3eb7b3d69063923060"} err="failed to get container status \"003caffe98d8d820c44300e9598a76a5d6760dc73e91cb3eb7b3d69063923060\": rpc error: code = NotFound desc = could not find container \"003caffe98d8d820c44300e9598a76a5d6760dc73e91cb3eb7b3d69063923060\": container with ID starting with 003caffe98d8d820c44300e9598a76a5d6760dc73e91cb3eb7b3d69063923060 not found: ID does not exist" Sep 11 14:42:21.912097 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:21.912063 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49eca948-49ea-4661-b3be-c98b9aa4cc16" path="/var/lib/kubelet/pods/49eca948-49ea-4661-b3be-c98b9aa4cc16/volumes" Sep 11 14:42:22.233729 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:22.233631 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:23.906177 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:23.906136 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:24.477098 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:24.477050 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:25.768677 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:25.768636 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:31.470465 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:31.470398 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:31.666579 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:31.666544 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:31.829825 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:31.829739 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq"] Sep 11 14:42:31.829989 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:31.829940 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" podUID="7444dcd1-ba89-4dd6-a013-c2fb82f0400a" containerName="istio-proxy" containerID="cri-o://8fae0b6ad3676bddbd35c686d04477c2f94134aa6a065cf0a70b444cad082114" gracePeriod=30 Sep 11 14:42:31.851805 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:31.851775 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:33.224543 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:33.224504 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:33.356329 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:33.356274 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:33.487193 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:33.487076 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:34.113513 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:34.113472 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:34.214580 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:34.214519 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:34.304451 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:34.304394 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:35.697692 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:35.697655 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:37.077380 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.077352 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:42:37.200521 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.200482 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-credential-socket\") pod \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " Sep 11 14:42:37.200709 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.200527 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-workload-socket\") pod \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " Sep 11 14:42:37.200709 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.200592 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-token\") pod \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " Sep 11 14:42:37.200709 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.200638 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-envoy\") pod \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " Sep 11 14:42:37.200709 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.200664 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-podinfo\") pod \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " Sep 11 14:42:37.200709 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.200690 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-workload-certs\") pod \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " Sep 11 14:42:37.201006 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.200785 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7444dcd1-ba89-4dd6-a013-c2fb82f0400a" (UID: "7444dcd1-ba89-4dd6-a013-c2fb82f0400a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:42:37.201006 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.200795 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2wfc\" (UniqueName: \"kubernetes.io/projected/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-kube-api-access-h2wfc\") pod \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " Sep 11 14:42:37.201006 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.200849 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istiod-ca-cert\") pod \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " Sep 11 14:42:37.201006 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.200884 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-data\") pod \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\" (UID: \"7444dcd1-ba89-4dd6-a013-c2fb82f0400a\") " Sep 11 14:42:37.201311 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.201267 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7444dcd1-ba89-4dd6-a013-c2fb82f0400a" (UID: "7444dcd1-ba89-4dd6-a013-c2fb82f0400a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:42:37.201389 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.201327 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-credential-socket\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:42:37.201657 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.201548 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-data" (OuterVolumeSpecName: "istio-data") pod "7444dcd1-ba89-4dd6-a013-c2fb82f0400a" (UID: "7444dcd1-ba89-4dd6-a013-c2fb82f0400a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:42:37.201657 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.201619 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7444dcd1-ba89-4dd6-a013-c2fb82f0400a" (UID: "7444dcd1-ba89-4dd6-a013-c2fb82f0400a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:42:37.201798 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.201748 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7444dcd1-ba89-4dd6-a013-c2fb82f0400a" (UID: "7444dcd1-ba89-4dd6-a013-c2fb82f0400a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:42:37.203398 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.203286 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-token" (OuterVolumeSpecName: "istio-token") pod "7444dcd1-ba89-4dd6-a013-c2fb82f0400a" (UID: "7444dcd1-ba89-4dd6-a013-c2fb82f0400a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:42:37.203398 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.203370 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-kube-api-access-h2wfc" (OuterVolumeSpecName: "kube-api-access-h2wfc") pod "7444dcd1-ba89-4dd6-a013-c2fb82f0400a" (UID: "7444dcd1-ba89-4dd6-a013-c2fb82f0400a"). InnerVolumeSpecName "kube-api-access-h2wfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:42:37.203656 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.203475 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7444dcd1-ba89-4dd6-a013-c2fb82f0400a" (UID: "7444dcd1-ba89-4dd6-a013-c2fb82f0400a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:42:37.203800 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.203765 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7444dcd1-ba89-4dd6-a013-c2fb82f0400a" (UID: "7444dcd1-ba89-4dd6-a013-c2fb82f0400a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:42:37.302763 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.302730 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-token\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:42:37.302763 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.302756 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-envoy\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:42:37.302763 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.302766 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-podinfo\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:42:37.302763 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.302774 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-workload-certs\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:42:37.303046 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.302783 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2wfc\" (UniqueName: \"kubernetes.io/projected/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-kube-api-access-h2wfc\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:42:37.303046 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.302793 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istiod-ca-cert\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:42:37.303046 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.302801 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-istio-data\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:42:37.303046 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.302809 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7444dcd1-ba89-4dd6-a013-c2fb82f0400a-workload-socket\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:42:37.733704 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.733673 2553 generic.go:356] "Generic (PLEG): container finished" podID="7444dcd1-ba89-4dd6-a013-c2fb82f0400a" containerID="8fae0b6ad3676bddbd35c686d04477c2f94134aa6a065cf0a70b444cad082114" exitCode=0 Sep 11 14:42:37.733881 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.733722 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" Sep 11 14:42:37.733881 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.733763 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" event={"ID":"7444dcd1-ba89-4dd6-a013-c2fb82f0400a","Type":"ContainerDied","Data":"8fae0b6ad3676bddbd35c686d04477c2f94134aa6a065cf0a70b444cad082114"} Sep 11 14:42:37.733881 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.733817 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq" event={"ID":"7444dcd1-ba89-4dd6-a013-c2fb82f0400a","Type":"ContainerDied","Data":"051c31f9b22b731ccf07022ac221d7d4f42c1a94684903b446a4439ffb30d376"} Sep 11 14:42:37.733881 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.733839 2553 scope.go:117] "RemoveContainer" containerID="8fae0b6ad3676bddbd35c686d04477c2f94134aa6a065cf0a70b444cad082114" Sep 11 14:42:37.743754 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.743737 2553 scope.go:117] "RemoveContainer" containerID="8bd7e8447f6b85d73b00d7f90ac4e548d476b48fbf6174d07d90974c0e6aa13f" Sep 11 14:42:37.751981 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.751964 2553 scope.go:117] "RemoveContainer" containerID="8fae0b6ad3676bddbd35c686d04477c2f94134aa6a065cf0a70b444cad082114" Sep 11 14:42:37.752261 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:42:37.752244 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fae0b6ad3676bddbd35c686d04477c2f94134aa6a065cf0a70b444cad082114\": container with ID starting with 8fae0b6ad3676bddbd35c686d04477c2f94134aa6a065cf0a70b444cad082114 not found: ID does not exist" containerID="8fae0b6ad3676bddbd35c686d04477c2f94134aa6a065cf0a70b444cad082114" Sep 11 14:42:37.752320 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.752269 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fae0b6ad3676bddbd35c686d04477c2f94134aa6a065cf0a70b444cad082114"} err="failed to get container status \"8fae0b6ad3676bddbd35c686d04477c2f94134aa6a065cf0a70b444cad082114\": rpc error: code = NotFound desc = could not find container \"8fae0b6ad3676bddbd35c686d04477c2f94134aa6a065cf0a70b444cad082114\": container with ID starting with 8fae0b6ad3676bddbd35c686d04477c2f94134aa6a065cf0a70b444cad082114 not found: ID does not exist" Sep 11 14:42:37.752320 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.752290 2553 scope.go:117] "RemoveContainer" containerID="8bd7e8447f6b85d73b00d7f90ac4e548d476b48fbf6174d07d90974c0e6aa13f" Sep 11 14:42:37.752591 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:42:37.752576 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd7e8447f6b85d73b00d7f90ac4e548d476b48fbf6174d07d90974c0e6aa13f\": container with ID starting with 8bd7e8447f6b85d73b00d7f90ac4e548d476b48fbf6174d07d90974c0e6aa13f not found: ID does not exist" containerID="8bd7e8447f6b85d73b00d7f90ac4e548d476b48fbf6174d07d90974c0e6aa13f" Sep 11 14:42:37.752645 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.752596 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd7e8447f6b85d73b00d7f90ac4e548d476b48fbf6174d07d90974c0e6aa13f"} err="failed to get container status \"8bd7e8447f6b85d73b00d7f90ac4e548d476b48fbf6174d07d90974c0e6aa13f\": rpc error: code = NotFound desc = could not find container \"8bd7e8447f6b85d73b00d7f90ac4e548d476b48fbf6174d07d90974c0e6aa13f\": container with ID starting with 8bd7e8447f6b85d73b00d7f90ac4e548d476b48fbf6174d07d90974c0e6aa13f not found: ID does not exist" Sep 11 14:42:37.759664 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.759638 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq"] Sep 11 14:42:37.771279 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.771253 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3893129d-openshift-default-569db84884-62pqq"] Sep 11 14:42:37.912667 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:37.912635 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7444dcd1-ba89-4dd6-a013-c2fb82f0400a" path="/var/lib/kubelet/pods/7444dcd1-ba89-4dd6-a013-c2fb82f0400a/volumes" Sep 11 14:42:42.098622 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:42.098561 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:42.190385 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:42.190344 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:42.818565 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:42.818530 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:49.117310 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:49.117266 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:49.218835 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:49.218799 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:49.322863 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:49.322832 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:50.135173 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:50.135142 2553 scope.go:117] "RemoveContainer" containerID="a9feb67961dfe33ff38a3f70f020c0d59f01fc85321f3a36cc5cf68f4b2b92dc" Sep 11 14:42:59.744113 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:59.744067 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:42:59.897557 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:42:59.897523 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:00.121223 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:00.121105 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:21.172528 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:21.172491 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:27.510130 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:27.510095 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:27.591599 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:27.591564 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:27.935050 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:27.935015 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:28.242756 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.242675 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx"] Sep 11 14:43:28.243077 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.243064 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49eca948-49ea-4661-b3be-c98b9aa4cc16" containerName="test-curl-7866-ba617a" Sep 11 14:43:28.243077 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.243078 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="49eca948-49ea-4661-b3be-c98b9aa4cc16" containerName="test-curl-7866-ba617a" Sep 11 14:43:28.243200 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.243090 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7444dcd1-ba89-4dd6-a013-c2fb82f0400a" containerName="istio-proxy" Sep 11 14:43:28.243200 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.243096 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7444dcd1-ba89-4dd6-a013-c2fb82f0400a" containerName="istio-proxy" Sep 11 14:43:28.243200 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.243111 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7444dcd1-ba89-4dd6-a013-c2fb82f0400a" containerName="istio-proxy" Sep 11 14:43:28.243200 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.243116 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7444dcd1-ba89-4dd6-a013-c2fb82f0400a" containerName="istio-proxy" Sep 11 14:43:28.243200 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.243172 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7444dcd1-ba89-4dd6-a013-c2fb82f0400a" containerName="istio-proxy" Sep 11 14:43:28.243200 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.243181 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7444dcd1-ba89-4dd6-a013-c2fb82f0400a" containerName="istio-proxy" Sep 11 14:43:28.243200 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.243188 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="49eca948-49ea-4661-b3be-c98b9aa4cc16" containerName="test-curl-7866-ba617a" Sep 11 14:43:28.246372 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.246354 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.250085 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.250064 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1af681-openshift-default-dockercfg-f4sxb\"" Sep 11 14:43:28.270013 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.269991 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx"] Sep 11 14:43:28.276838 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.276811 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:28.281172 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.281137 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8259fc17-5f91-43bb-81da-fb291f67424c-istiod-ca-cert\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.281297 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.281177 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8259fc17-5f91-43bb-81da-fb291f67424c-istio-token\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.281369 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.281282 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-workload-certs\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.281369 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.281346 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-istio-envoy\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.281483 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.281381 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-istio-data\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.281483 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.281463 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8259fc17-5f91-43bb-81da-fb291f67424c-istio-podinfo\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.281606 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.281500 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-credential-socket\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.281606 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.281531 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-workload-socket\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.281606 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.281558 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ffzd\" (UniqueName: \"kubernetes.io/projected/8259fc17-5f91-43bb-81da-fb291f67424c-kube-api-access-5ffzd\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.382326 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.382285 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8259fc17-5f91-43bb-81da-fb291f67424c-istiod-ca-cert\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.382579 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.382335 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8259fc17-5f91-43bb-81da-fb291f67424c-istio-token\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.382579 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.382460 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-workload-certs\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.382579 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.382503 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-istio-envoy\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.382579 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.382540 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-istio-data\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.382579 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.382569 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8259fc17-5f91-43bb-81da-fb291f67424c-istio-podinfo\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.382908 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.382614 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-credential-socket\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.382908 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.382660 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-workload-socket\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.382908 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.382691 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ffzd\" (UniqueName: \"kubernetes.io/projected/8259fc17-5f91-43bb-81da-fb291f67424c-kube-api-access-5ffzd\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.382908 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.382866 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-workload-certs\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.383137 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.382951 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-istio-data\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.383137 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.383111 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-credential-socket\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.383219 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.383163 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8259fc17-5f91-43bb-81da-fb291f67424c-istiod-ca-cert\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.383219 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.383190 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-workload-socket\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.384997 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.384976 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-istio-envoy\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.385165 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.385144 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8259fc17-5f91-43bb-81da-fb291f67424c-istio-podinfo\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.397495 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.397465 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8259fc17-5f91-43bb-81da-fb291f67424c-istio-token\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.397673 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.397651 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ffzd\" (UniqueName: \"kubernetes.io/projected/8259fc17-5f91-43bb-81da-fb291f67424c-kube-api-access-5ffzd\") pod \"e2e-api-a-1af681-openshift-default-6896b46554-8zqbx\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.556947 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.556860 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:28.578039 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.578005 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:28.840365 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.840338 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx"] Sep 11 14:43:28.842394 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:43:28.842359 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8259fc17_5f91_43bb_81da_fb291f67424c.slice/crio-bc7b242b362430383f50736a4e777677041d9817a8ff2c334fa3079c3cf68daa WatchSource:0}: Error finding container bc7b242b362430383f50736a4e777677041d9817a8ff2c334fa3079c3cf68daa: Status 404 returned error can't find the container with id bc7b242b362430383f50736a4e777677041d9817a8ff2c334fa3079c3cf68daa Sep 11 14:43:28.844621 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.844587 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:43:28.844700 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.844671 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:43:28.844739 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.844707 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:43:28.997481 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.997448 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" event={"ID":"8259fc17-5f91-43bb-81da-fb291f67424c","Type":"ContainerStarted","Data":"be78b8ab48225851e495ae76b6f832b3bdfb4e1fa9dff2a103bae02004a1e607"} Sep 11 14:43:28.997669 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:28.997501 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" event={"ID":"8259fc17-5f91-43bb-81da-fb291f67424c","Type":"ContainerStarted","Data":"bc7b242b362430383f50736a4e777677041d9817a8ff2c334fa3079c3cf68daa"} Sep 11 14:43:29.037130 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:29.037077 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" podStartSLOduration=1.037059138 podStartE2EDuration="1.037059138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:43:29.036280072 +0000 UTC m=+1719.650386417" watchObservedRunningTime="2026-09-11 14:43:29.037059138 +0000 UTC m=+1719.651165482" Sep 11 14:43:29.557230 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:29.557181 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:29.562402 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:29.562380 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:30.002789 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:30.002755 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:30.003624 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:30.003607 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:30.391105 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:30.391072 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:31.064203 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:31.064162 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:31.151565 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:31.151525 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:31.250499 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:31.250456 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:32.316634 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:32.316590 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:33.770401 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:33.770368 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:35.260539 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:35.260503 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:36.323081 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:36.323041 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:36.454340 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:36.454300 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:36.558959 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:36.558927 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:36.962853 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:36.962817 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:37.069531 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:37.069496 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:37.183039 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:37.183009 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p"] Sep 11 14:43:37.186934 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:37.186911 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p" Sep 11 14:43:37.190328 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:37.190306 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1af681\"" Sep 11 14:43:37.190493 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:37.190380 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1af681-dockercfg-td9k6\"" Sep 11 14:43:37.200234 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:37.200212 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p"] Sep 11 14:43:37.271362 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:37.271255 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6396676-bcf9-4948-815d-3b61dc20e771-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p\" (UID: \"c6396676-bcf9-4948-815d-3b61dc20e771\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p" Sep 11 14:43:37.271362 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:37.271308 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pbhz\" (UniqueName: \"kubernetes.io/projected/c6396676-bcf9-4948-815d-3b61dc20e771-kube-api-access-9pbhz\") pod \"payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p\" (UID: \"c6396676-bcf9-4948-815d-3b61dc20e771\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p" Sep 11 14:43:37.372837 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:37.372800 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6396676-bcf9-4948-815d-3b61dc20e771-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p\" (UID: \"c6396676-bcf9-4948-815d-3b61dc20e771\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p" Sep 11 14:43:37.372837 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:37.372835 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pbhz\" (UniqueName: \"kubernetes.io/projected/c6396676-bcf9-4948-815d-3b61dc20e771-kube-api-access-9pbhz\") pod \"payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p\" (UID: \"c6396676-bcf9-4948-815d-3b61dc20e771\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p" Sep 11 14:43:37.373497 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:37.373479 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6396676-bcf9-4948-815d-3b61dc20e771-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p\" (UID: \"c6396676-bcf9-4948-815d-3b61dc20e771\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p" Sep 11 14:43:37.382168 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:37.382147 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pbhz\" (UniqueName: \"kubernetes.io/projected/c6396676-bcf9-4948-815d-3b61dc20e771-kube-api-access-9pbhz\") pod \"payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p\" (UID: \"c6396676-bcf9-4948-815d-3b61dc20e771\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p" Sep 11 14:43:37.498440 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:37.498376 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p" Sep 11 14:43:37.772980 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:37.772958 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p"] Sep 11 14:43:38.051608 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:38.051572 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p" event={"ID":"c6396676-bcf9-4948-815d-3b61dc20e771","Type":"ContainerStarted","Data":"703ca8782ee1422d9dda008e07538e41120f9a344ebfbfaf1add8857bae74749"} Sep 11 14:43:38.051803 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:38.051629 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p" event={"ID":"c6396676-bcf9-4948-815d-3b61dc20e771","Type":"ContainerStarted","Data":"fbe55a0a30e3a1802990605da95279819cdb408d3ce5cec713a22ef31304235b"} Sep 11 14:43:38.051803 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:38.051675 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p" Sep 11 14:43:38.086529 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:38.086428 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p" podStartSLOduration=1.086394371 podStartE2EDuration="1.086394371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:43:38.08447931 +0000 UTC m=+1728.698585657" watchObservedRunningTime="2026-09-11 14:43:38.086394371 +0000 UTC m=+1728.700500717" Sep 11 14:43:46.330376 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:46.330315 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:49.059238 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:49.059197 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p" Sep 11 14:43:50.171269 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:50.171223 2553 scope.go:117] "RemoveContainer" containerID="daad23456631abc835fac1561575ddd82f0e49d4fab340c9d4f378079287e8a5" Sep 11 14:43:54.133838 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:54.133803 2553 generic.go:356] "Generic (PLEG): container finished" podID="8259fc17-5f91-43bb-81da-fb291f67424c" containerID="be78b8ab48225851e495ae76b6f832b3bdfb4e1fa9dff2a103bae02004a1e607" exitCode=0 Sep 11 14:43:54.134255 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:54.133879 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" event={"ID":"8259fc17-5f91-43bb-81da-fb291f67424c","Type":"ContainerDied","Data":"be78b8ab48225851e495ae76b6f832b3bdfb4e1fa9dff2a103bae02004a1e607"} Sep 11 14:43:54.134310 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:54.134297 2553 scope.go:117] "RemoveContainer" containerID="be78b8ab48225851e495ae76b6f832b3bdfb4e1fa9dff2a103bae02004a1e607" Sep 11 14:43:54.135400 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:54.135374 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:43:54.135470 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:54.135457 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:43:54.135515 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:54.135504 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:43:55.141126 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:55.141093 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" event={"ID":"8259fc17-5f91-43bb-81da-fb291f67424c","Type":"ContainerStarted","Data":"e0b94cd51a3f0b7e0afd41653b4c53f971edc1262065cd98d5b1f931cad3552a"} Sep 11 14:43:55.557117 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:55.557007 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:43:55.558650 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:55.558616 2553 patch_prober.go:28] interesting pod/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 11 14:43:55.558796 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:55.558696 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" podUID="8259fc17-5f91-43bb-81da-fb291f67424c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 11 14:43:56.558408 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:56.558357 2553 patch_prober.go:28] interesting pod/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 11 14:43:56.558926 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:56.558463 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" podUID="8259fc17-5f91-43bb-81da-fb291f67424c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 11 14:43:57.558273 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:57.558233 2553 patch_prober.go:28] interesting pod/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 11 14:43:57.558504 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:57.558305 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" podUID="8259fc17-5f91-43bb-81da-fb291f67424c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 11 14:43:57.665206 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:57.665165 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:58.557530 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:58.557491 2553 patch_prober.go:28] interesting pod/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 11 14:43:58.557823 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:58.557792 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" podUID="8259fc17-5f91-43bb-81da-fb291f67424c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 11 14:43:59.361459 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:59.361401 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:43:59.557695 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:59.557655 2553 patch_prober.go:28] interesting pod/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Sep 11 14:43:59.557903 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:43:59.557731 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" podUID="8259fc17-5f91-43bb-81da-fb291f67424c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Sep 11 14:44:00.003020 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:00.002979 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:44:00.371915 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:00.371825 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:00.561465 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:00.561405 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:44:00.562538 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:00.562518 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:44:01.310053 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:01.310013 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:01.734877 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:01.734835 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:01.841235 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:01.841197 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:02.200797 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.200766 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9"] Sep 11 14:44:02.204588 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.204570 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.208379 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.208354 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-fc74dd-openshift-default-dockercfg-ppb95\"" Sep 11 14:44:02.224209 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.224185 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-podinfo\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.224922 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.224316 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-envoy\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.224922 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.224350 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldbds\" (UniqueName: \"kubernetes.io/projected/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-kube-api-access-ldbds\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.224922 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.224401 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-credential-socket\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.224922 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.224472 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-data\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.224922 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.224495 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istiod-ca-cert\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.224922 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.224530 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-workload-socket\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.224922 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.224590 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-workload-certs\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.224922 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.224613 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-token\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.227052 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.227033 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9"] Sep 11 14:44:02.264969 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.264940 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:02.325575 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.325533 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-workload-socket\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.325784 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.325613 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-workload-certs\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.325784 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.325639 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-token\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.325784 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.325713 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-podinfo\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.325784 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.325755 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-envoy\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.325784 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.325782 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldbds\" (UniqueName: \"kubernetes.io/projected/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-kube-api-access-ldbds\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.326074 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.325832 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-credential-socket\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.326074 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.325881 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-data\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.326074 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.325908 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istiod-ca-cert\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.326074 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.325989 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-workload-socket\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.326653 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.326626 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istiod-ca-cert\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.326774 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.326681 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-credential-socket\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.326894 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.326875 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-data\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.327026 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.326945 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-workload-certs\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.329967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.329938 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-envoy\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.331723 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.331700 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-podinfo\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.342505 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.342454 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldbds\" (UniqueName: \"kubernetes.io/projected/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-kube-api-access-ldbds\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.343242 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.343216 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-token\") pod \"e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.518327 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.518225 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:02.554374 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.554333 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:02.781543 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.781463 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:02.820000 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.819968 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9"] Sep 11 14:44:02.823456 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:44:02.823428 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df4c5e2_7ec2_4073_ae5a_075cdcb95c07.slice/crio-f3c82c3066b43da4527eca2560c1ad6cef2adac344c9aa204be609b1815c3626 WatchSource:0}: Error finding container f3c82c3066b43da4527eca2560c1ad6cef2adac344c9aa204be609b1815c3626: Status 404 returned error can't find the container with id f3c82c3066b43da4527eca2560c1ad6cef2adac344c9aa204be609b1815c3626 Sep 11 14:44:02.825762 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.825729 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:44:02.825857 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.825796 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:44:02.825857 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:02.825827 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:44:03.196991 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:03.196950 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" event={"ID":"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07","Type":"ContainerStarted","Data":"eb8d8355ee5bab977080c5d07e9acd6b7d7a4b74189c782d29587e79746201ae"} Sep 11 14:44:03.197207 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:03.197007 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" event={"ID":"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07","Type":"ContainerStarted","Data":"f3c82c3066b43da4527eca2560c1ad6cef2adac344c9aa204be609b1815c3626"} Sep 11 14:44:03.233778 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:03.233726 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" podStartSLOduration=1.233707898 podStartE2EDuration="1.233707898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:44:03.230192844 +0000 UTC m=+1753.844299188" watchObservedRunningTime="2026-09-11 14:44:03.233707898 +0000 UTC m=+1753.847814241" Sep 11 14:44:03.518780 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:03.518688 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:03.524099 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:03.524073 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:04.041627 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:04.041592 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:04.202647 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:04.202611 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:04.203672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:04.203647 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:05.447072 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:05.447039 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:05.548785 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:05.548745 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:07.185975 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:07.185935 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:07.308746 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:07.308709 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:07.433629 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:07.433598 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:08.281284 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:08.281246 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:08.417642 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:08.417610 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:09.157153 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:09.157111 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:10.480304 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:10.480273 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:11.942288 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:11.942254 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:13.208606 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:13.208570 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:14.229116 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:14.229085 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:15.350317 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:15.350281 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:17.846544 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:17.846493 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:18.910478 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:18.910441 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:19.791296 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:19.791254 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:20.137872 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:20.137852 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:21.160239 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:21.160201 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:22.472879 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:22.472841 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:23.382030 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:23.382000 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1af681-lq9xm"] Sep 11 14:44:23.385518 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:23.385493 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1af681-lq9xm" Sep 11 14:44:23.391229 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:23.391207 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:44:23.391350 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:23.391237 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6975v\"" Sep 11 14:44:23.407629 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:23.407600 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1af681-lq9xm"] Sep 11 14:44:23.480020 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:23.479985 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:23.534018 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:23.533984 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1dd540f-6e69-4316-8c8c-6877fd12815a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1af681-lq9xm\" (UID: \"e1dd540f-6e69-4316-8c8c-6877fd12815a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1af681-lq9xm" Sep 11 14:44:23.634645 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:23.634559 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1dd540f-6e69-4316-8c8c-6877fd12815a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1af681-lq9xm\" (UID: \"e1dd540f-6e69-4316-8c8c-6877fd12815a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1af681-lq9xm" Sep 11 14:44:23.635184 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:23.635164 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1dd540f-6e69-4316-8c8c-6877fd12815a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1af681-lq9xm\" (UID: \"e1dd540f-6e69-4316-8c8c-6877fd12815a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1af681-lq9xm" Sep 11 14:44:23.696251 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:23.696216 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1af681-lq9xm" Sep 11 14:44:23.999124 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:23.999098 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1af681-lq9xm"] Sep 11 14:44:24.000759 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:44:24.000730 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1dd540f_6e69_4316_8c8c_6877fd12815a.slice/crio-488cc6c3f3045a79505262e06277cebe7604ce8924e332e1d252c4501ea0d630 WatchSource:0}: Error finding container 488cc6c3f3045a79505262e06277cebe7604ce8924e332e1d252c4501ea0d630: Status 404 returned error can't find the container with id 488cc6c3f3045a79505262e06277cebe7604ce8924e332e1d252c4501ea0d630 Sep 11 14:44:24.306278 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:24.306239 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1af681-lq9xm" event={"ID":"e1dd540f-6e69-4316-8c8c-6877fd12815a","Type":"ContainerStarted","Data":"65d8bd6c6cc2c1af6a7d8c8bcbd502cdd18d13d7c436d3c1b6bdd975834e49f1"} Sep 11 14:44:24.306456 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:24.306297 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1af681-lq9xm" event={"ID":"e1dd540f-6e69-4316-8c8c-6877fd12815a","Type":"ContainerStarted","Data":"488cc6c3f3045a79505262e06277cebe7604ce8924e332e1d252c4501ea0d630"} Sep 11 14:44:24.331612 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:24.331553 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1af681-lq9xm" podStartSLOduration=1.331535869 podStartE2EDuration="1.331535869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:44:24.329618576 +0000 UTC m=+1774.943724922" watchObservedRunningTime="2026-09-11 14:44:24.331535869 +0000 UTC m=+1774.945642243" Sep 11 14:44:24.520732 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:24.520632 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:25.314475 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:25.314431 2553 generic.go:356] "Generic (PLEG): container finished" podID="e1dd540f-6e69-4316-8c8c-6877fd12815a" containerID="65d8bd6c6cc2c1af6a7d8c8bcbd502cdd18d13d7c436d3c1b6bdd975834e49f1" exitCode=0 Sep 11 14:44:25.314684 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:25.314510 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1af681-lq9xm" event={"ID":"e1dd540f-6e69-4316-8c8c-6877fd12815a","Type":"ContainerDied","Data":"65d8bd6c6cc2c1af6a7d8c8bcbd502cdd18d13d7c436d3c1b6bdd975834e49f1"} Sep 11 14:44:25.584997 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:25.584917 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:26.468753 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:26.468730 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1af681-lq9xm" Sep 11 14:44:26.564756 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:26.564725 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1dd540f-6e69-4316-8c8c-6877fd12815a-maas-api-service-ca\") pod \"e1dd540f-6e69-4316-8c8c-6877fd12815a\" (UID: \"e1dd540f-6e69-4316-8c8c-6877fd12815a\") " Sep 11 14:44:26.565059 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:26.565037 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1dd540f-6e69-4316-8c8c-6877fd12815a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e1dd540f-6e69-4316-8c8c-6877fd12815a" (UID: "e1dd540f-6e69-4316-8c8c-6877fd12815a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:44:26.665921 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:26.665886 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1dd540f-6e69-4316-8c8c-6877fd12815a-maas-api-service-ca\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:44:26.725842 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:26.725810 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:27.326565 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:27.326457 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1af681-lq9xm" event={"ID":"e1dd540f-6e69-4316-8c8c-6877fd12815a","Type":"ContainerDied","Data":"488cc6c3f3045a79505262e06277cebe7604ce8924e332e1d252c4501ea0d630"} Sep 11 14:44:27.326565 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:27.326480 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1af681-lq9xm" Sep 11 14:44:27.326565 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:27.326543 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="488cc6c3f3045a79505262e06277cebe7604ce8924e332e1d252c4501ea0d630" Sep 11 14:44:27.328425 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:27.328372 2553 generic.go:356] "Generic (PLEG): container finished" podID="7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" containerID="eb8d8355ee5bab977080c5d07e9acd6b7d7a4b74189c782d29587e79746201ae" exitCode=0 Sep 11 14:44:27.328554 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:27.328438 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" event={"ID":"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07","Type":"ContainerDied","Data":"eb8d8355ee5bab977080c5d07e9acd6b7d7a4b74189c782d29587e79746201ae"} Sep 11 14:44:27.328801 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:27.328789 2553 scope.go:117] "RemoveContainer" containerID="eb8d8355ee5bab977080c5d07e9acd6b7d7a4b74189c782d29587e79746201ae" Sep 11 14:44:27.329855 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:27.329827 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:44:27.329930 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:27.329891 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:44:27.329930 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:27.329923 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:44:27.593635 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:27.593533 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:28.335643 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:28.335613 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" event={"ID":"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07","Type":"ContainerStarted","Data":"8dcd41634b5dd88a83c4ebcb13f12df8b50cb86647a002b0ca2c5603c3867f83"} Sep 11 14:44:28.390785 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:28.390747 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:28.519076 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:28.519043 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:28.519363 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:28.519341 2553 patch_prober.go:28] interesting pod/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 14:44:28.519439 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:28.519392 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" podUID="7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 14:44:29.373873 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:29.373838 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:29.519534 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:29.519496 2553 patch_prober.go:28] interesting pod/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 14:44:29.519719 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:29.519548 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" podUID="7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 14:44:30.366077 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:30.365861 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:30.519584 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:30.519547 2553 patch_prober.go:28] interesting pod/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 14:44:30.520000 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:30.519601 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" podUID="7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 14:44:31.518722 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:31.518689 2553 patch_prober.go:28] interesting pod/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 14:44:31.518911 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:31.518739 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" podUID="7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 14:44:31.557332 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:31.557292 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:32.519382 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:32.519347 2553 patch_prober.go:28] interesting pod/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 14:44:32.519592 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:32.519402 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" podUID="7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 14:44:32.733111 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:32.733073 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7866-0c2da6"] Sep 11 14:44:32.733725 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:32.733705 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1dd540f-6e69-4316-8c8c-6877fd12815a" containerName="revoke-keys" Sep 11 14:44:32.733814 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:32.733727 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1dd540f-6e69-4316-8c8c-6877fd12815a" containerName="revoke-keys" Sep 11 14:44:32.733875 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:32.733856 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1dd540f-6e69-4316-8c8c-6877fd12815a" containerName="revoke-keys" Sep 11 14:44:32.736075 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:32.736054 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7866-0c2da6" Sep 11 14:44:32.739705 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:32.739689 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vdvtq\"" Sep 11 14:44:32.746031 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:32.746008 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7866-0c2da6"] Sep 11 14:44:32.829913 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:32.829815 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6gtg\" (UniqueName: \"kubernetes.io/projected/0a09086f-466e-4a29-afd0-af246041e602-kube-api-access-v6gtg\") pod \"test-curl-7866-0c2da6\" (UID: \"0a09086f-466e-4a29-afd0-af246041e602\") " pod="opendatahub/test-curl-7866-0c2da6" Sep 11 14:44:32.930498 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:32.930464 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6gtg\" (UniqueName: \"kubernetes.io/projected/0a09086f-466e-4a29-afd0-af246041e602-kube-api-access-v6gtg\") pod \"test-curl-7866-0c2da6\" (UID: \"0a09086f-466e-4a29-afd0-af246041e602\") " pod="opendatahub/test-curl-7866-0c2da6" Sep 11 14:44:32.940630 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:32.940597 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6gtg\" (UniqueName: \"kubernetes.io/projected/0a09086f-466e-4a29-afd0-af246041e602-kube-api-access-v6gtg\") pod \"test-curl-7866-0c2da6\" (UID: \"0a09086f-466e-4a29-afd0-af246041e602\") " pod="opendatahub/test-curl-7866-0c2da6" Sep 11 14:44:33.046487 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:33.046452 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7866-0c2da6" Sep 11 14:44:33.331406 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:33.331384 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7866-0c2da6"] Sep 11 14:44:33.332855 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:44:33.332828 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a09086f_466e_4a29_afd0_af246041e602.slice/crio-73cbeb535500c2fad6d2424e703e386e0282c296282ed1985ae3cf683f1e210c WatchSource:0}: Error finding container 73cbeb535500c2fad6d2424e703e386e0282c296282ed1985ae3cf683f1e210c: Status 404 returned error can't find the container with id 73cbeb535500c2fad6d2424e703e386e0282c296282ed1985ae3cf683f1e210c Sep 11 14:44:33.375137 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:33.375077 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7866-0c2da6" event={"ID":"0a09086f-466e-4a29-afd0-af246041e602","Type":"ContainerStarted","Data":"73cbeb535500c2fad6d2424e703e386e0282c296282ed1985ae3cf683f1e210c"} Sep 11 14:44:33.518765 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:33.518728 2553 patch_prober.go:28] interesting pod/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Sep 11 14:44:33.518947 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:33.518786 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" podUID="7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Sep 11 14:44:33.529663 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:33.529630 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:34.203528 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:34.203496 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:34.381510 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:34.381474 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7866-0c2da6" event={"ID":"0a09086f-466e-4a29-afd0-af246041e602","Type":"ContainerStarted","Data":"7dc2c85fb1361f85b0dab08ff54609002657af14dcb03e9280c19e507329ee11"} Sep 11 14:44:34.402072 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:34.402023 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7866-0c2da6" podStartSLOduration=2.402007311 podStartE2EDuration="2.402007311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:44:34.40036812 +0000 UTC m=+1785.014474461" watchObservedRunningTime="2026-09-11 14:44:34.402007311 +0000 UTC m=+1785.016113655" Sep 11 14:44:34.523213 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:34.523135 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:34.523975 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:34.523957 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:44:35.064219 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:35.064183 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7866-0c2da6"] Sep 11 14:44:35.069953 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:35.069920 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7866-0c2da6"] Sep 11 14:44:35.199538 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:35.199509 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7866-d6ba17"] Sep 11 14:44:35.199934 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:35.199920 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a09086f-466e-4a29-afd0-af246041e602" containerName="test-curl-7866-0c2da6" Sep 11 14:44:35.199980 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:35.199935 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a09086f-466e-4a29-afd0-af246041e602" containerName="test-curl-7866-0c2da6" Sep 11 14:44:35.200013 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:35.200004 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a09086f-466e-4a29-afd0-af246041e602" containerName="test-curl-7866-0c2da6" Sep 11 14:44:35.202230 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:35.202216 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7866-d6ba17" Sep 11 14:44:35.225733 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:35.225703 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7866-d6ba17"] Sep 11 14:44:35.354718 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:35.354680 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwjct\" (UniqueName: \"kubernetes.io/projected/05feda4c-61e8-4d72-919a-5e9c7b0b2dc6-kube-api-access-xwjct\") pod \"test-curl-7866-d6ba17\" (UID: \"05feda4c-61e8-4d72-919a-5e9c7b0b2dc6\") " pod="opendatahub/test-curl-7866-d6ba17" Sep 11 14:44:35.387587 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:44:35.387557 2553 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7866-0c2da6" container="test-curl-7866-0c2da6" Sep 11 14:44:35.455471 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:35.455366 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwjct\" (UniqueName: \"kubernetes.io/projected/05feda4c-61e8-4d72-919a-5e9c7b0b2dc6-kube-api-access-xwjct\") pod \"test-curl-7866-d6ba17\" (UID: \"05feda4c-61e8-4d72-919a-5e9c7b0b2dc6\") " pod="opendatahub/test-curl-7866-d6ba17" Sep 11 14:44:35.464807 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:35.464775 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwjct\" (UniqueName: \"kubernetes.io/projected/05feda4c-61e8-4d72-919a-5e9c7b0b2dc6-kube-api-access-xwjct\") pod \"test-curl-7866-d6ba17\" (UID: \"05feda4c-61e8-4d72-919a-5e9c7b0b2dc6\") " pod="opendatahub/test-curl-7866-d6ba17" Sep 11 14:44:35.511623 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:35.511590 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7866-d6ba17" Sep 11 14:44:35.797045 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:35.797018 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7866-d6ba17"] Sep 11 14:44:35.799347 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:44:35.799317 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05feda4c_61e8_4d72_919a_5e9c7b0b2dc6.slice/crio-92a80cdc81e37088f2eadabceb7fa4c822ed432deca7dbe2952d60c6295b41f3 WatchSource:0}: Error finding container 92a80cdc81e37088f2eadabceb7fa4c822ed432deca7dbe2952d60c6295b41f3: Status 404 returned error can't find the container with id 92a80cdc81e37088f2eadabceb7fa4c822ed432deca7dbe2952d60c6295b41f3 Sep 11 14:44:36.393711 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:36.393673 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7866-d6ba17" event={"ID":"05feda4c-61e8-4d72-919a-5e9c7b0b2dc6","Type":"ContainerStarted","Data":"d62f36d53758b8d4798134b6e8baf0d32f793aecda8b05eeef7dad7d92fa7df3"} Sep 11 14:44:36.393711 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:36.393716 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7866-d6ba17" event={"ID":"05feda4c-61e8-4d72-919a-5e9c7b0b2dc6","Type":"ContainerStarted","Data":"92a80cdc81e37088f2eadabceb7fa4c822ed432deca7dbe2952d60c6295b41f3"} Sep 11 14:44:36.394211 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:36.393778 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7866-0c2da6" podUID="0a09086f-466e-4a29-afd0-af246041e602" containerName="test-curl-7866-0c2da6" containerID="cri-o://7dc2c85fb1361f85b0dab08ff54609002657af14dcb03e9280c19e507329ee11" gracePeriod=2 Sep 11 14:44:36.412686 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:36.412639 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7866-d6ba17" podStartSLOduration=1.412623575 podStartE2EDuration="1.412623575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:44:36.411710752 +0000 UTC m=+1787.025817097" watchObservedRunningTime="2026-09-11 14:44:36.412623575 +0000 UTC m=+1787.026729928" Sep 11 14:44:37.087429 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:37.087392 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7866-d6ba17"] Sep 11 14:44:37.092678 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:37.092642 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7866-d6ba17"] Sep 11 14:44:37.399161 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:44:37.399133 2553 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7866-d6ba17" container="test-curl-7866-d6ba17" Sep 11 14:44:38.404806 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:38.404775 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7866-d6ba17" podUID="05feda4c-61e8-4d72-919a-5e9c7b0b2dc6" containerName="test-curl-7866-d6ba17" containerID="cri-o://d62f36d53758b8d4798134b6e8baf0d32f793aecda8b05eeef7dad7d92fa7df3" gracePeriod=2 Sep 11 14:44:38.538544 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:38.538519 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7866-0c2da6" Sep 11 14:44:38.584214 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:38.584181 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6gtg\" (UniqueName: \"kubernetes.io/projected/0a09086f-466e-4a29-afd0-af246041e602-kube-api-access-v6gtg\") pod \"0a09086f-466e-4a29-afd0-af246041e602\" (UID: \"0a09086f-466e-4a29-afd0-af246041e602\") " Sep 11 14:44:38.586227 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:38.586205 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a09086f-466e-4a29-afd0-af246041e602-kube-api-access-v6gtg" (OuterVolumeSpecName: "kube-api-access-v6gtg") pod "0a09086f-466e-4a29-afd0-af246041e602" (UID: "0a09086f-466e-4a29-afd0-af246041e602"). InnerVolumeSpecName "kube-api-access-v6gtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:38.685570 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:38.685493 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6gtg\" (UniqueName: \"kubernetes.io/projected/0a09086f-466e-4a29-afd0-af246041e602-kube-api-access-v6gtg\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:44:39.308979 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.308947 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:39.372825 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.372787 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p"] Sep 11 14:44:39.373075 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.373047 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p" podUID="c6396676-bcf9-4948-815d-3b61dc20e771" containerName="payload-pre-processing" containerID="cri-o://703ca8782ee1422d9dda008e07538e41120f9a344ebfbfaf1add8857bae74749" gracePeriod=30 Sep 11 14:44:39.405882 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.405833 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:39.411698 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.411670 2553 generic.go:356] "Generic (PLEG): container finished" podID="0a09086f-466e-4a29-afd0-af246041e602" containerID="7dc2c85fb1361f85b0dab08ff54609002657af14dcb03e9280c19e507329ee11" exitCode=137 Sep 11 14:44:39.411867 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.411720 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7866-0c2da6" Sep 11 14:44:39.411867 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.411767 2553 scope.go:117] "RemoveContainer" containerID="7dc2c85fb1361f85b0dab08ff54609002657af14dcb03e9280c19e507329ee11" Sep 11 14:44:39.476463 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:44:39.476434 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7866-0c2da6\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="0a09086f-466e-4a29-afd0-af246041e602" pod="opendatahub/test-curl-7866-0c2da6" Sep 11 14:44:39.485564 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.485541 2553 scope.go:117] "RemoveContainer" containerID="7dc2c85fb1361f85b0dab08ff54609002657af14dcb03e9280c19e507329ee11" Sep 11 14:44:39.485984 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:44:39.485962 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc2c85fb1361f85b0dab08ff54609002657af14dcb03e9280c19e507329ee11\": container with ID starting with 7dc2c85fb1361f85b0dab08ff54609002657af14dcb03e9280c19e507329ee11 not found: ID does not exist" containerID="7dc2c85fb1361f85b0dab08ff54609002657af14dcb03e9280c19e507329ee11" Sep 11 14:44:39.486056 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.485996 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc2c85fb1361f85b0dab08ff54609002657af14dcb03e9280c19e507329ee11"} err="failed to get container status \"7dc2c85fb1361f85b0dab08ff54609002657af14dcb03e9280c19e507329ee11\": rpc error: code = NotFound desc = could not find container \"7dc2c85fb1361f85b0dab08ff54609002657af14dcb03e9280c19e507329ee11\": container with ID starting with 7dc2c85fb1361f85b0dab08ff54609002657af14dcb03e9280c19e507329ee11 not found: ID does not exist" Sep 11 14:44:39.511010 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.510978 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:39.606939 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.606860 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1af681-lq9xm"] Sep 11 14:44:39.611588 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.611546 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:39.612256 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.612234 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1af681-lq9xm"] Sep 11 14:44:39.633299 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.633279 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p" Sep 11 14:44:39.636030 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:44:39.636004 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7866-0c2da6\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="0a09086f-466e-4a29-afd0-af246041e602" pod="opendatahub/test-curl-7866-0c2da6" Sep 11 14:44:39.694651 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.694622 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6396676-bcf9-4948-815d-3b61dc20e771-plugins-config-volume\") pod \"c6396676-bcf9-4948-815d-3b61dc20e771\" (UID: \"c6396676-bcf9-4948-815d-3b61dc20e771\") " Sep 11 14:44:39.694863 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.694749 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pbhz\" (UniqueName: \"kubernetes.io/projected/c6396676-bcf9-4948-815d-3b61dc20e771-kube-api-access-9pbhz\") pod \"c6396676-bcf9-4948-815d-3b61dc20e771\" (UID: \"c6396676-bcf9-4948-815d-3b61dc20e771\") " Sep 11 14:44:39.695159 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.695036 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6396676-bcf9-4948-815d-3b61dc20e771-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c6396676-bcf9-4948-815d-3b61dc20e771" (UID: "c6396676-bcf9-4948-815d-3b61dc20e771"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:44:39.697024 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.696995 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6396676-bcf9-4948-815d-3b61dc20e771-kube-api-access-9pbhz" (OuterVolumeSpecName: "kube-api-access-9pbhz") pod "c6396676-bcf9-4948-815d-3b61dc20e771" (UID: "c6396676-bcf9-4948-815d-3b61dc20e771"). InnerVolumeSpecName "kube-api-access-9pbhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:39.701994 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.701964 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:39.796034 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.795999 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pbhz\" (UniqueName: \"kubernetes.io/projected/c6396676-bcf9-4948-815d-3b61dc20e771-kube-api-access-9pbhz\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:44:39.796034 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.796027 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6396676-bcf9-4948-815d-3b61dc20e771-plugins-config-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:44:39.916255 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:44:39.916200 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7866-0c2da6\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="0a09086f-466e-4a29-afd0-af246041e602" pod="opendatahub/test-curl-7866-0c2da6" Sep 11 14:44:39.918364 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.918332 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a09086f-466e-4a29-afd0-af246041e602" path="/var/lib/kubelet/pods/0a09086f-466e-4a29-afd0-af246041e602/volumes" Sep 11 14:44:39.918794 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.918775 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1dd540f-6e69-4316-8c8c-6877fd12815a" path="/var/lib/kubelet/pods/e1dd540f-6e69-4316-8c8c-6877fd12815a/volumes" Sep 11 14:44:39.998404 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:39.998369 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:40.101583 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:40.101545 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:40.420118 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:40.420092 2553 generic.go:356] "Generic (PLEG): container finished" podID="c6396676-bcf9-4948-815d-3b61dc20e771" containerID="703ca8782ee1422d9dda008e07538e41120f9a344ebfbfaf1add8857bae74749" exitCode=0 Sep 11 14:44:40.420494 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:40.420138 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p" event={"ID":"c6396676-bcf9-4948-815d-3b61dc20e771","Type":"ContainerDied","Data":"703ca8782ee1422d9dda008e07538e41120f9a344ebfbfaf1add8857bae74749"} Sep 11 14:44:40.420494 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:40.420149 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p" Sep 11 14:44:40.420494 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:40.420193 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p" event={"ID":"c6396676-bcf9-4948-815d-3b61dc20e771","Type":"ContainerDied","Data":"fbe55a0a30e3a1802990605da95279819cdb408d3ce5cec713a22ef31304235b"} Sep 11 14:44:40.420494 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:40.420228 2553 scope.go:117] "RemoveContainer" containerID="703ca8782ee1422d9dda008e07538e41120f9a344ebfbfaf1add8857bae74749" Sep 11 14:44:40.464115 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:40.464091 2553 scope.go:117] "RemoveContainer" containerID="703ca8782ee1422d9dda008e07538e41120f9a344ebfbfaf1add8857bae74749" Sep 11 14:44:40.464449 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:44:40.464409 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"703ca8782ee1422d9dda008e07538e41120f9a344ebfbfaf1add8857bae74749\": container with ID starting with 703ca8782ee1422d9dda008e07538e41120f9a344ebfbfaf1add8857bae74749 not found: ID does not exist" containerID="703ca8782ee1422d9dda008e07538e41120f9a344ebfbfaf1add8857bae74749" Sep 11 14:44:40.464559 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:40.464458 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"703ca8782ee1422d9dda008e07538e41120f9a344ebfbfaf1add8857bae74749"} err="failed to get container status \"703ca8782ee1422d9dda008e07538e41120f9a344ebfbfaf1add8857bae74749\": rpc error: code = NotFound desc = could not find container \"703ca8782ee1422d9dda008e07538e41120f9a344ebfbfaf1add8857bae74749\": container with ID starting with 703ca8782ee1422d9dda008e07538e41120f9a344ebfbfaf1add8857bae74749 not found: ID does not exist" Sep 11 14:44:40.472762 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:40.472739 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p"] Sep 11 14:44:40.479546 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:40.479523 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1af681-695784cb66-b6r7p"] Sep 11 14:44:40.590662 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:40.590640 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7866-d6ba17" Sep 11 14:44:40.603951 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:40.603930 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwjct\" (UniqueName: \"kubernetes.io/projected/05feda4c-61e8-4d72-919a-5e9c7b0b2dc6-kube-api-access-xwjct\") pod \"05feda4c-61e8-4d72-919a-5e9c7b0b2dc6\" (UID: \"05feda4c-61e8-4d72-919a-5e9c7b0b2dc6\") " Sep 11 14:44:40.605939 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:40.605919 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05feda4c-61e8-4d72-919a-5e9c7b0b2dc6-kube-api-access-xwjct" (OuterVolumeSpecName: "kube-api-access-xwjct") pod "05feda4c-61e8-4d72-919a-5e9c7b0b2dc6" (UID: "05feda4c-61e8-4d72-919a-5e9c7b0b2dc6"). InnerVolumeSpecName "kube-api-access-xwjct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:40.705460 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:40.705347 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwjct\" (UniqueName: \"kubernetes.io/projected/05feda4c-61e8-4d72-919a-5e9c7b0b2dc6-kube-api-access-xwjct\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:44:41.122398 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:41.122309 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:41.426460 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:41.426403 2553 generic.go:356] "Generic (PLEG): container finished" podID="05feda4c-61e8-4d72-919a-5e9c7b0b2dc6" containerID="d62f36d53758b8d4798134b6e8baf0d32f793aecda8b05eeef7dad7d92fa7df3" exitCode=137 Sep 11 14:44:41.426460 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:41.426455 2553 scope.go:117] "RemoveContainer" containerID="d62f36d53758b8d4798134b6e8baf0d32f793aecda8b05eeef7dad7d92fa7df3" Sep 11 14:44:41.426973 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:41.426479 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7866-d6ba17" Sep 11 14:44:41.435035 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:41.435021 2553 scope.go:117] "RemoveContainer" containerID="d62f36d53758b8d4798134b6e8baf0d32f793aecda8b05eeef7dad7d92fa7df3" Sep 11 14:44:41.435322 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:44:41.435303 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d62f36d53758b8d4798134b6e8baf0d32f793aecda8b05eeef7dad7d92fa7df3\": container with ID starting with d62f36d53758b8d4798134b6e8baf0d32f793aecda8b05eeef7dad7d92fa7df3 not found: ID does not exist" containerID="d62f36d53758b8d4798134b6e8baf0d32f793aecda8b05eeef7dad7d92fa7df3" Sep 11 14:44:41.435394 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:41.435335 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62f36d53758b8d4798134b6e8baf0d32f793aecda8b05eeef7dad7d92fa7df3"} err="failed to get container status \"d62f36d53758b8d4798134b6e8baf0d32f793aecda8b05eeef7dad7d92fa7df3\": rpc error: code = NotFound desc = could not find container \"d62f36d53758b8d4798134b6e8baf0d32f793aecda8b05eeef7dad7d92fa7df3\": container with ID starting with d62f36d53758b8d4798134b6e8baf0d32f793aecda8b05eeef7dad7d92fa7df3 not found: ID does not exist" Sep 11 14:44:41.438148 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:44:41.438125 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7866-d6ba17\" is forbidden: User \"system:node:ip-10-0-135-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-17.ec2.internal' and this object" podUID="05feda4c-61e8-4d72-919a-5e9c7b0b2dc6" pod="opendatahub/test-curl-7866-d6ba17" Sep 11 14:44:41.875921 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:41.875820 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:41.912496 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:41.912468 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05feda4c-61e8-4d72-919a-5e9c7b0b2dc6" path="/var/lib/kubelet/pods/05feda4c-61e8-4d72-919a-5e9c7b0b2dc6/volumes" Sep 11 14:44:41.912890 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:41.912873 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6396676-bcf9-4948-815d-3b61dc20e771" path="/var/lib/kubelet/pods/c6396676-bcf9-4948-815d-3b61dc20e771/volumes" Sep 11 14:44:42.145057 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:42.145008 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:42.936513 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:42.936481 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:44.005461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:44.005402 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:44.898183 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:44.898152 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:46.012840 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.012810 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh"] Sep 11 14:44:46.013245 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.013231 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6396676-bcf9-4948-815d-3b61dc20e771" containerName="payload-pre-processing" Sep 11 14:44:46.013245 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.013246 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6396676-bcf9-4948-815d-3b61dc20e771" containerName="payload-pre-processing" Sep 11 14:44:46.013340 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.013263 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05feda4c-61e8-4d72-919a-5e9c7b0b2dc6" containerName="test-curl-7866-d6ba17" Sep 11 14:44:46.013340 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.013269 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="05feda4c-61e8-4d72-919a-5e9c7b0b2dc6" containerName="test-curl-7866-d6ba17" Sep 11 14:44:46.013404 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.013350 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6396676-bcf9-4948-815d-3b61dc20e771" containerName="payload-pre-processing" Sep 11 14:44:46.013404 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.013358 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="05feda4c-61e8-4d72-919a-5e9c7b0b2dc6" containerName="test-curl-7866-d6ba17" Sep 11 14:44:46.016236 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.016220 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:46.019740 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.019717 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8feb68\"/\"kube-root-ca.crt\"" Sep 11 14:44:46.021214 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.021197 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-8feb68\"/\"openshift-service-ca.crt\"" Sep 11 14:44:46.021393 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.021373 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8feb68\"/\"rate-test-model-8feb68-kserve-self-signed-certs\"" Sep 11 14:44:46.021455 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.021394 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-8feb68\"/\"default-dockercfg-4bqkh\"" Sep 11 14:44:46.027733 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.027710 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh"] Sep 11 14:44:46.043609 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.043581 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:46.057961 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.057939 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-tmp-dir\") pod \"rate-test-model-8feb68-kserve-68895c5675-mjwmh\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:46.058059 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.057982 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-home\") pod \"rate-test-model-8feb68-kserve-68895c5675-mjwmh\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:46.058059 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.058018 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcd7w\" (UniqueName: \"kubernetes.io/projected/c42f0609-3b37-4823-a21e-854e2d0ed6ac-kube-api-access-bcd7w\") pod \"rate-test-model-8feb68-kserve-68895c5675-mjwmh\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:46.058059 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.058049 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-model-cache\") pod \"rate-test-model-8feb68-kserve-68895c5675-mjwmh\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:46.058201 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.058084 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42f0609-3b37-4823-a21e-854e2d0ed6ac-tls-certs\") pod \"rate-test-model-8feb68-kserve-68895c5675-mjwmh\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:46.058201 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.058150 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-dshm\") pod \"rate-test-model-8feb68-kserve-68895c5675-mjwmh\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:46.159632 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.159598 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-dshm\") pod \"rate-test-model-8feb68-kserve-68895c5675-mjwmh\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:46.159842 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.159663 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-tmp-dir\") pod \"rate-test-model-8feb68-kserve-68895c5675-mjwmh\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:46.159842 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.159770 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-home\") pod \"rate-test-model-8feb68-kserve-68895c5675-mjwmh\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:46.159842 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.159806 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcd7w\" (UniqueName: \"kubernetes.io/projected/c42f0609-3b37-4823-a21e-854e2d0ed6ac-kube-api-access-bcd7w\") pod \"rate-test-model-8feb68-kserve-68895c5675-mjwmh\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:46.160209 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.159879 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-model-cache\") pod \"rate-test-model-8feb68-kserve-68895c5675-mjwmh\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:46.160209 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.159922 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42f0609-3b37-4823-a21e-854e2d0ed6ac-tls-certs\") pod \"rate-test-model-8feb68-kserve-68895c5675-mjwmh\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:46.160311 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.160236 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-tmp-dir\") pod \"rate-test-model-8feb68-kserve-68895c5675-mjwmh\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:46.160311 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.160262 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-home\") pod \"rate-test-model-8feb68-kserve-68895c5675-mjwmh\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:46.160441 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.160303 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-model-cache\") pod \"rate-test-model-8feb68-kserve-68895c5675-mjwmh\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:46.162402 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.162380 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-dshm\") pod \"rate-test-model-8feb68-kserve-68895c5675-mjwmh\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:46.162676 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.162655 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42f0609-3b37-4823-a21e-854e2d0ed6ac-tls-certs\") pod \"rate-test-model-8feb68-kserve-68895c5675-mjwmh\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:46.171037 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.171012 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcd7w\" (UniqueName: \"kubernetes.io/projected/c42f0609-3b37-4823-a21e-854e2d0ed6ac-kube-api-access-bcd7w\") pod \"rate-test-model-8feb68-kserve-68895c5675-mjwmh\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:46.325369 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.325277 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:46.604377 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.604347 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh"] Sep 11 14:44:46.607188 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:44:46.607161 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc42f0609_3b37_4823_a21e_854e2d0ed6ac.slice/crio-1acbe24dc35b6fae86aafd088b24600f6cb9aa4bcf34ad9405a1282552d6d1e0 WatchSource:0}: Error finding container 1acbe24dc35b6fae86aafd088b24600f6cb9aa4bcf34ad9405a1282552d6d1e0: Status 404 returned error can't find the container with id 1acbe24dc35b6fae86aafd088b24600f6cb9aa4bcf34ad9405a1282552d6d1e0 Sep 11 14:44:46.994231 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:46.994197 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:47.110058 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:47.110012 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:47.306511 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:47.306425 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx"] Sep 11 14:44:47.306687 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:47.306665 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" podUID="8259fc17-5f91-43bb-81da-fb291f67424c" containerName="istio-proxy" containerID="cri-o://e0b94cd51a3f0b7e0afd41653b4c53f971edc1262065cd98d5b1f931cad3552a" gracePeriod=30 Sep 11 14:44:47.350124 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:47.350086 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:47.463869 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:47.463834 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" event={"ID":"c42f0609-3b37-4823-a21e-854e2d0ed6ac","Type":"ContainerStarted","Data":"47cfc3951391027fafe4ec473b8098009d7bee0caf1f297258a6da5d8d9f7a34"} Sep 11 14:44:47.463869 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:47.463881 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" event={"ID":"c42f0609-3b37-4823-a21e-854e2d0ed6ac","Type":"ContainerStarted","Data":"1acbe24dc35b6fae86aafd088b24600f6cb9aa4bcf34ad9405a1282552d6d1e0"} Sep 11 14:44:47.496154 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:47.496098 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" podStartSLOduration=2.496081488 podStartE2EDuration="2.496081488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:44:47.493550429 +0000 UTC m=+1798.107656786" watchObservedRunningTime="2026-09-11 14:44:47.496081488 +0000 UTC m=+1798.110187889" Sep 11 14:44:48.015990 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:48.015960 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:49.120212 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:49.120177 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:49.917635 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:49.917601 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:50.000753 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:50.000718 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmqjh_6b2452d9-1730-458c-b380-dbcfd9e5be19/ovn-acl-logging/0.log" Sep 11 14:44:50.007684 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:50.007663 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmqjh_6b2452d9-1730-458c-b380-dbcfd9e5be19/ovn-acl-logging/0.log" Sep 11 14:44:50.192682 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:50.192602 2553 scope.go:117] "RemoveContainer" containerID="fbd71b86c7476df4755dab457e0f89ecc37158b5af0dccabd56d272a86f7ffd1" Sep 11 14:44:50.917010 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:50.916970 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:51.942949 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:51.942904 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:52.494669 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.494581 2553 generic.go:356] "Generic (PLEG): container finished" podID="8259fc17-5f91-43bb-81da-fb291f67424c" containerID="e0b94cd51a3f0b7e0afd41653b4c53f971edc1262065cd98d5b1f931cad3552a" exitCode=0 Sep 11 14:44:52.494835 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.494741 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" event={"ID":"8259fc17-5f91-43bb-81da-fb291f67424c","Type":"ContainerDied","Data":"e0b94cd51a3f0b7e0afd41653b4c53f971edc1262065cd98d5b1f931cad3552a"} Sep 11 14:44:52.494909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.494863 2553 scope.go:117] "RemoveContainer" containerID="be78b8ab48225851e495ae76b6f832b3bdfb4e1fa9dff2a103bae02004a1e607" Sep 11 14:44:52.568714 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.568636 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:44:52.624121 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.624094 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-istio-envoy\") pod \"8259fc17-5f91-43bb-81da-fb291f67424c\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " Sep 11 14:44:52.624320 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.624137 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-workload-socket\") pod \"8259fc17-5f91-43bb-81da-fb291f67424c\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " Sep 11 14:44:52.624320 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.624169 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8259fc17-5f91-43bb-81da-fb291f67424c-istiod-ca-cert\") pod \"8259fc17-5f91-43bb-81da-fb291f67424c\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " Sep 11 14:44:52.624320 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.624189 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-istio-data\") pod \"8259fc17-5f91-43bb-81da-fb291f67424c\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " Sep 11 14:44:52.624320 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.624237 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-workload-certs\") pod \"8259fc17-5f91-43bb-81da-fb291f67424c\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " Sep 11 14:44:52.624320 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.624277 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ffzd\" (UniqueName: \"kubernetes.io/projected/8259fc17-5f91-43bb-81da-fb291f67424c-kube-api-access-5ffzd\") pod \"8259fc17-5f91-43bb-81da-fb291f67424c\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " Sep 11 14:44:52.624320 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.624320 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8259fc17-5f91-43bb-81da-fb291f67424c-istio-token\") pod \"8259fc17-5f91-43bb-81da-fb291f67424c\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " Sep 11 14:44:52.624677 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.624392 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8259fc17-5f91-43bb-81da-fb291f67424c-istio-podinfo\") pod \"8259fc17-5f91-43bb-81da-fb291f67424c\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " Sep 11 14:44:52.624677 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.624447 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-credential-socket\") pod \"8259fc17-5f91-43bb-81da-fb291f67424c\" (UID: \"8259fc17-5f91-43bb-81da-fb291f67424c\") " Sep 11 14:44:52.624677 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.624505 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8259fc17-5f91-43bb-81da-fb291f67424c" (UID: "8259fc17-5f91-43bb-81da-fb291f67424c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:52.624677 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.624595 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8259fc17-5f91-43bb-81da-fb291f67424c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8259fc17-5f91-43bb-81da-fb291f67424c" (UID: "8259fc17-5f91-43bb-81da-fb291f67424c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:44:52.624880 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.624726 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8259fc17-5f91-43bb-81da-fb291f67424c" (UID: "8259fc17-5f91-43bb-81da-fb291f67424c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:52.624880 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.624766 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8259fc17-5f91-43bb-81da-fb291f67424c" (UID: "8259fc17-5f91-43bb-81da-fb291f67424c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:52.624977 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.624896 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-workload-socket\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:44:52.624977 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.624908 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8259fc17-5f91-43bb-81da-fb291f67424c-istiod-ca-cert\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:44:52.624977 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.624917 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-workload-certs\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:44:52.624977 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.624926 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-credential-socket\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:44:52.625175 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.625089 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-istio-data" (OuterVolumeSpecName: "istio-data") pod "8259fc17-5f91-43bb-81da-fb291f67424c" (UID: "8259fc17-5f91-43bb-81da-fb291f67424c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:52.626606 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.626578 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8259fc17-5f91-43bb-81da-fb291f67424c" (UID: "8259fc17-5f91-43bb-81da-fb291f67424c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:44:52.626860 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.626832 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8259fc17-5f91-43bb-81da-fb291f67424c-kube-api-access-5ffzd" (OuterVolumeSpecName: "kube-api-access-5ffzd") pod "8259fc17-5f91-43bb-81da-fb291f67424c" (UID: "8259fc17-5f91-43bb-81da-fb291f67424c"). InnerVolumeSpecName "kube-api-access-5ffzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:52.627094 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.627062 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8259fc17-5f91-43bb-81da-fb291f67424c-istio-token" (OuterVolumeSpecName: "istio-token") pod "8259fc17-5f91-43bb-81da-fb291f67424c" (UID: "8259fc17-5f91-43bb-81da-fb291f67424c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:44:52.627193 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.627151 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8259fc17-5f91-43bb-81da-fb291f67424c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8259fc17-5f91-43bb-81da-fb291f67424c" (UID: "8259fc17-5f91-43bb-81da-fb291f67424c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:44:52.725944 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.725895 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-istio-data\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:44:52.725944 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.725930 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ffzd\" (UniqueName: \"kubernetes.io/projected/8259fc17-5f91-43bb-81da-fb291f67424c-kube-api-access-5ffzd\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:44:52.725944 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.725942 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8259fc17-5f91-43bb-81da-fb291f67424c-istio-token\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:44:52.725944 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.725950 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8259fc17-5f91-43bb-81da-fb291f67424c-istio-podinfo\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:44:52.725944 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:52.725958 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8259fc17-5f91-43bb-81da-fb291f67424c-istio-envoy\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:44:53.066014 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:53.065984 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:53.501672 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:53.501637 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" Sep 11 14:44:53.501857 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:53.501680 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx" event={"ID":"8259fc17-5f91-43bb-81da-fb291f67424c","Type":"ContainerDied","Data":"bc7b242b362430383f50736a4e777677041d9817a8ff2c334fa3079c3cf68daa"} Sep 11 14:44:53.501857 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:53.501753 2553 scope.go:117] "RemoveContainer" containerID="e0b94cd51a3f0b7e0afd41653b4c53f971edc1262065cd98d5b1f931cad3552a" Sep 11 14:44:53.529139 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:53.529112 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx"] Sep 11 14:44:53.533993 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:53.533967 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1af681-openshift-default-6896b46554-8zqbx"] Sep 11 14:44:53.912593 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:53.912555 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8259fc17-5f91-43bb-81da-fb291f67424c" path="/var/lib/kubelet/pods/8259fc17-5f91-43bb-81da-fb291f67424c/volumes" Sep 11 14:44:54.107707 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:54.107673 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:44:56.326484 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:56.326448 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:56.326882 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:56.326497 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:56.338583 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:56.338554 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:44:56.528439 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:44:56.528400 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:45:00.254572 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:00.254539 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-977c12-29818965-kdz4b"] Sep 11 14:45:00.255021 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:00.255004 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8259fc17-5f91-43bb-81da-fb291f67424c" containerName="istio-proxy" Sep 11 14:45:00.255075 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:00.255022 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8259fc17-5f91-43bb-81da-fb291f67424c" containerName="istio-proxy" Sep 11 14:45:00.255075 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:00.255032 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8259fc17-5f91-43bb-81da-fb291f67424c" containerName="istio-proxy" Sep 11 14:45:00.255075 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:00.255038 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8259fc17-5f91-43bb-81da-fb291f67424c" containerName="istio-proxy" Sep 11 14:45:00.255188 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:00.255123 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8259fc17-5f91-43bb-81da-fb291f67424c" containerName="istio-proxy" Sep 11 14:45:00.255188 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:00.255132 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8259fc17-5f91-43bb-81da-fb291f67424c" containerName="istio-proxy" Sep 11 14:45:00.259797 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:00.259777 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8feb68-29818965-jtqsz"] Sep 11 14:45:00.259909 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:00.259887 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-977c12-29818965-kdz4b" Sep 11 14:45:00.263191 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:00.263173 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28858c-29818965-rpfjq"] Sep 11 14:45:00.263282 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:00.263262 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8feb68-29818965-jtqsz" Sep 11 14:45:00.266324 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:00.266305 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6975v\"" Sep 11 14:45:00.266860 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:00.266841 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28858c-29818965-rpfjq" Sep 11 14:45:00.271507 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:00.271491 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-977c12-29818965-kdz4b" Sep 11 14:45:00.280313 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:00.280292 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8feb68-29818965-jtqsz" Sep 11 14:45:00.281538 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:00.281497 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-977c12-29818965-kdz4b"] Sep 11 14:45:00.301003 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:00.300970 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28858c-29818965-rpfjq"] Sep 11 14:45:00.321691 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:00.321636 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8feb68-29818965-jtqsz"] Sep 11 14:45:00.327342 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:00.326716 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28858c-29818965-rpfjq" Sep 11 14:45:00.578576 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:00.578496 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-977c12-29818965-kdz4b"] Sep 11 14:45:00.583144 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:45:00.583118 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a47a90_8e1d_4d0f_9aa9_2fb80b977711.slice/crio-f10fd34948e2d8f252ddbbbbc9f2622c4b5282756285ee5090f742a77259c8c5 WatchSource:0}: Error finding container f10fd34948e2d8f252ddbbbbc9f2622c4b5282756285ee5090f742a77259c8c5: Status 404 returned error can't find the container with id f10fd34948e2d8f252ddbbbbc9f2622c4b5282756285ee5090f742a77259c8c5 Sep 11 14:45:00.592465 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:00.592441 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8feb68-29818965-jtqsz"] Sep 11 14:45:00.595909 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:45:00.595889 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd45bc23_fd9d_4dd0_bb7f_d824fee14fbe.slice/crio-0d94dc946eed79eaa2583a7a60820b53a57f858db96068e61b9061b44f62a1ed WatchSource:0}: Error finding container 0d94dc946eed79eaa2583a7a60820b53a57f858db96068e61b9061b44f62a1ed: Status 404 returned error can't find the container with id 0d94dc946eed79eaa2583a7a60820b53a57f858db96068e61b9061b44f62a1ed Sep 11 14:45:00.641704 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:00.641681 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28858c-29818965-rpfjq"] Sep 11 14:45:00.643144 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:45:00.643117 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac00535_28d3_48d3_99a1_7571db396d08.slice/crio-07c3036c34aead553522c9d9ca52e1eff661d550bc4a8c17de528b385c1db9f8 WatchSource:0}: Error finding container 07c3036c34aead553522c9d9ca52e1eff661d550bc4a8c17de528b385c1db9f8: Status 404 returned error can't find the container with id 07c3036c34aead553522c9d9ca52e1eff661d550bc4a8c17de528b385c1db9f8 Sep 11 14:45:01.544228 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:01.544184 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28858c-29818965-rpfjq" event={"ID":"5ac00535-28d3-48d3-99a1-7571db396d08","Type":"ContainerStarted","Data":"2c00ab6043d36a902daca61cffa880683061a8da389419a6850bb2dff30f43bb"} Sep 11 14:45:01.544732 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:01.544256 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28858c-29818965-rpfjq" event={"ID":"5ac00535-28d3-48d3-99a1-7571db396d08","Type":"ContainerStarted","Data":"07c3036c34aead553522c9d9ca52e1eff661d550bc4a8c17de528b385c1db9f8"} Sep 11 14:45:01.546458 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:01.546402 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8feb68-29818965-jtqsz" event={"ID":"cd45bc23-fd9d-4dd0-bb7f-d824fee14fbe","Type":"ContainerStarted","Data":"c883c73f430bf9ed5677b9eb04427c06a516085b58dc5438c0dfb921904f839c"} Sep 11 14:45:01.546714 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:01.546686 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8feb68-29818965-jtqsz" event={"ID":"cd45bc23-fd9d-4dd0-bb7f-d824fee14fbe","Type":"ContainerStarted","Data":"0d94dc946eed79eaa2583a7a60820b53a57f858db96068e61b9061b44f62a1ed"} Sep 11 14:45:01.548630 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:01.548602 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-977c12-29818965-kdz4b" event={"ID":"74a47a90-8e1d-4d0f-9aa9-2fb80b977711","Type":"ContainerStarted","Data":"142ef67051fa296482b4e6536e1995eeba1c8e62d63ee65b190de25484af33b9"} Sep 11 14:45:01.548736 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:01.548649 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-977c12-29818965-kdz4b" event={"ID":"74a47a90-8e1d-4d0f-9aa9-2fb80b977711","Type":"ContainerStarted","Data":"f10fd34948e2d8f252ddbbbbc9f2622c4b5282756285ee5090f742a77259c8c5"} Sep 11 14:45:01.570574 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:01.570527 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28858c-29818965-rpfjq" podStartSLOduration=1.5705137900000001 podStartE2EDuration="1.57051379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:45:01.568374716 +0000 UTC m=+1812.182481059" watchObservedRunningTime="2026-09-11 14:45:01.57051379 +0000 UTC m=+1812.184620127" Sep 11 14:45:01.593123 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:01.593076 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8feb68-29818965-jtqsz" podStartSLOduration=1.5930635180000001 podStartE2EDuration="1.593063518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:45:01.591402605 +0000 UTC m=+1812.205509159" watchObservedRunningTime="2026-09-11 14:45:01.593063518 +0000 UTC m=+1812.207169904" Sep 11 14:45:01.613854 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:01.613805 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-977c12-29818965-kdz4b" podStartSLOduration=1.613792589 podStartE2EDuration="1.613792589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:45:01.611242724 +0000 UTC m=+1812.225349067" watchObservedRunningTime="2026-09-11 14:45:01.613792589 +0000 UTC m=+1812.227898990" Sep 11 14:45:02.556036 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:02.555998 2553 generic.go:356] "Generic (PLEG): container finished" podID="5ac00535-28d3-48d3-99a1-7571db396d08" containerID="2c00ab6043d36a902daca61cffa880683061a8da389419a6850bb2dff30f43bb" exitCode=0 Sep 11 14:45:02.556559 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:02.556078 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28858c-29818965-rpfjq" event={"ID":"5ac00535-28d3-48d3-99a1-7571db396d08","Type":"ContainerDied","Data":"2c00ab6043d36a902daca61cffa880683061a8da389419a6850bb2dff30f43bb"} Sep 11 14:45:02.557723 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:02.557698 2553 generic.go:356] "Generic (PLEG): container finished" podID="cd45bc23-fd9d-4dd0-bb7f-d824fee14fbe" containerID="c883c73f430bf9ed5677b9eb04427c06a516085b58dc5438c0dfb921904f839c" exitCode=0 Sep 11 14:45:02.557878 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:02.557776 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8feb68-29818965-jtqsz" event={"ID":"cd45bc23-fd9d-4dd0-bb7f-d824fee14fbe","Type":"ContainerDied","Data":"c883c73f430bf9ed5677b9eb04427c06a516085b58dc5438c0dfb921904f839c"} Sep 11 14:45:02.559357 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:02.559331 2553 generic.go:356] "Generic (PLEG): container finished" podID="74a47a90-8e1d-4d0f-9aa9-2fb80b977711" containerID="142ef67051fa296482b4e6536e1995eeba1c8e62d63ee65b190de25484af33b9" exitCode=0 Sep 11 14:45:02.559490 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:02.559404 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-977c12-29818965-kdz4b" event={"ID":"74a47a90-8e1d-4d0f-9aa9-2fb80b977711","Type":"ContainerDied","Data":"142ef67051fa296482b4e6536e1995eeba1c8e62d63ee65b190de25484af33b9"} Sep 11 14:45:03.389633 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:03.389601 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:03.569129 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:03.569084 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:03.702073 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:03.701972 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:03.722827 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:03.722787 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-977c12-29818965-kdz4b" Sep 11 14:45:03.748035 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:03.748005 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28858c-29818965-rpfjq" Sep 11 14:45:03.774701 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:03.774675 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8feb68-29818965-jtqsz" Sep 11 14:45:03.833112 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:03.832799 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:04.230306 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:04.230272 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:04.356566 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:04.356529 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:04.570085 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:04.569987 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28858c-29818965-rpfjq" Sep 11 14:45:04.570085 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:04.569982 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28858c-29818965-rpfjq" event={"ID":"5ac00535-28d3-48d3-99a1-7571db396d08","Type":"ContainerDied","Data":"07c3036c34aead553522c9d9ca52e1eff661d550bc4a8c17de528b385c1db9f8"} Sep 11 14:45:04.570644 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:04.570137 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07c3036c34aead553522c9d9ca52e1eff661d550bc4a8c17de528b385c1db9f8" Sep 11 14:45:04.571598 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:04.571577 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8feb68-29818965-jtqsz" Sep 11 14:45:04.571729 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:04.571599 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8feb68-29818965-jtqsz" event={"ID":"cd45bc23-fd9d-4dd0-bb7f-d824fee14fbe","Type":"ContainerDied","Data":"0d94dc946eed79eaa2583a7a60820b53a57f858db96068e61b9061b44f62a1ed"} Sep 11 14:45:04.571729 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:04.571640 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d94dc946eed79eaa2583a7a60820b53a57f858db96068e61b9061b44f62a1ed" Sep 11 14:45:04.573061 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:04.573045 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-977c12-29818965-kdz4b" Sep 11 14:45:04.573180 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:04.573099 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-977c12-29818965-kdz4b" event={"ID":"74a47a90-8e1d-4d0f-9aa9-2fb80b977711","Type":"ContainerDied","Data":"f10fd34948e2d8f252ddbbbbc9f2622c4b5282756285ee5090f742a77259c8c5"} Sep 11 14:45:04.573180 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:04.573150 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f10fd34948e2d8f252ddbbbbc9f2622c4b5282756285ee5090f742a77259c8c5" Sep 11 14:45:05.591331 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:05.591295 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:06.644502 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:06.644469 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:07.855485 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:07.855447 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:07.951604 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:07.951567 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:08.735469 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:08.735406 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:10.635322 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:10.635290 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:10.730570 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:10.730532 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:10.945862 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:10.945808 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9"] Sep 11 14:45:10.946123 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:10.946075 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" podUID="7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" containerName="istio-proxy" containerID="cri-o://8dcd41634b5dd88a83c4ebcb13f12df8b50cb86647a002b0ca2c5603c3867f83" gracePeriod=30 Sep 11 14:45:10.978842 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:10.978803 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:11.648180 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:11.648143 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:12.737970 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:12.737933 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:13.963461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:13.963402 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:14.992501 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:14.992455 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:16.178651 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.178619 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:16.205553 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.205524 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:45:16.269323 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.269244 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-workload-certs\") pod \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " Sep 11 14:45:16.269323 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.269290 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-data\") pod \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " Sep 11 14:45:16.269562 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.269328 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldbds\" (UniqueName: \"kubernetes.io/projected/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-kube-api-access-ldbds\") pod \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " Sep 11 14:45:16.269562 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.269401 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-podinfo\") pod \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " Sep 11 14:45:16.269562 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.269444 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-envoy\") pod \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " Sep 11 14:45:16.269562 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.269475 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-workload-socket\") pod \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " Sep 11 14:45:16.269562 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.269516 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-credential-socket\") pod \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " Sep 11 14:45:16.269824 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.269552 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" (UID: "7df4c5e2-7ec2-4073-ae5a-075cdcb95c07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:45:16.269824 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.269589 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-token\") pod \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " Sep 11 14:45:16.269824 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.269616 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istiod-ca-cert\") pod \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\" (UID: \"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07\") " Sep 11 14:45:16.269824 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.269635 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-data" (OuterVolumeSpecName: "istio-data") pod "7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" (UID: "7df4c5e2-7ec2-4073-ae5a-075cdcb95c07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:45:16.269824 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.269778 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" (UID: "7df4c5e2-7ec2-4073-ae5a-075cdcb95c07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:45:16.270106 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.269837 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" (UID: "7df4c5e2-7ec2-4073-ae5a-075cdcb95c07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:45:16.270106 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.270003 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-workload-socket\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:45:16.270106 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.270021 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-credential-socket\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:45:16.270106 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.270038 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-workload-certs\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:45:16.270106 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.270051 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-data\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:45:16.270106 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.270072 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" (UID: "7df4c5e2-7ec2-4073-ae5a-075cdcb95c07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:45:16.271769 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.271732 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-kube-api-access-ldbds" (OuterVolumeSpecName: "kube-api-access-ldbds") pod "7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" (UID: "7df4c5e2-7ec2-4073-ae5a-075cdcb95c07"). InnerVolumeSpecName "kube-api-access-ldbds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:45:16.271887 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.271814 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" (UID: "7df4c5e2-7ec2-4073-ae5a-075cdcb95c07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:45:16.271945 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.271880 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-token" (OuterVolumeSpecName: "istio-token") pod "7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" (UID: "7df4c5e2-7ec2-4073-ae5a-075cdcb95c07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:45:16.272033 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.272018 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" (UID: "7df4c5e2-7ec2-4073-ae5a-075cdcb95c07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:45:16.371471 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.371408 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-podinfo\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:45:16.371471 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.371461 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-envoy\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:45:16.371471 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.371470 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istio-token\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:45:16.371471 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.371478 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-istiod-ca-cert\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:45:16.371753 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.371487 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldbds\" (UniqueName: \"kubernetes.io/projected/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07-kube-api-access-ldbds\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:45:16.639928 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.639893 2553 generic.go:356] "Generic (PLEG): container finished" podID="7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" containerID="8dcd41634b5dd88a83c4ebcb13f12df8b50cb86647a002b0ca2c5603c3867f83" exitCode=0 Sep 11 14:45:16.640089 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.639944 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" Sep 11 14:45:16.640089 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.639957 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" event={"ID":"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07","Type":"ContainerDied","Data":"8dcd41634b5dd88a83c4ebcb13f12df8b50cb86647a002b0ca2c5603c3867f83"} Sep 11 14:45:16.640089 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.640008 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9" event={"ID":"7df4c5e2-7ec2-4073-ae5a-075cdcb95c07","Type":"ContainerDied","Data":"f3c82c3066b43da4527eca2560c1ad6cef2adac344c9aa204be609b1815c3626"} Sep 11 14:45:16.640089 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.640040 2553 scope.go:117] "RemoveContainer" containerID="8dcd41634b5dd88a83c4ebcb13f12df8b50cb86647a002b0ca2c5603c3867f83" Sep 11 14:45:16.650917 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.650896 2553 scope.go:117] "RemoveContainer" containerID="eb8d8355ee5bab977080c5d07e9acd6b7d7a4b74189c782d29587e79746201ae" Sep 11 14:45:16.659448 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.659427 2553 scope.go:117] "RemoveContainer" containerID="8dcd41634b5dd88a83c4ebcb13f12df8b50cb86647a002b0ca2c5603c3867f83" Sep 11 14:45:16.659698 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:45:16.659678 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dcd41634b5dd88a83c4ebcb13f12df8b50cb86647a002b0ca2c5603c3867f83\": container with ID starting with 8dcd41634b5dd88a83c4ebcb13f12df8b50cb86647a002b0ca2c5603c3867f83 not found: ID does not exist" containerID="8dcd41634b5dd88a83c4ebcb13f12df8b50cb86647a002b0ca2c5603c3867f83" Sep 11 14:45:16.659738 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.659707 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dcd41634b5dd88a83c4ebcb13f12df8b50cb86647a002b0ca2c5603c3867f83"} err="failed to get container status \"8dcd41634b5dd88a83c4ebcb13f12df8b50cb86647a002b0ca2c5603c3867f83\": rpc error: code = NotFound desc = could not find container \"8dcd41634b5dd88a83c4ebcb13f12df8b50cb86647a002b0ca2c5603c3867f83\": container with ID starting with 8dcd41634b5dd88a83c4ebcb13f12df8b50cb86647a002b0ca2c5603c3867f83 not found: ID does not exist" Sep 11 14:45:16.659738 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.659727 2553 scope.go:117] "RemoveContainer" containerID="eb8d8355ee5bab977080c5d07e9acd6b7d7a4b74189c782d29587e79746201ae" Sep 11 14:45:16.659959 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:45:16.659943 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb8d8355ee5bab977080c5d07e9acd6b7d7a4b74189c782d29587e79746201ae\": container with ID starting with eb8d8355ee5bab977080c5d07e9acd6b7d7a4b74189c782d29587e79746201ae not found: ID does not exist" containerID="eb8d8355ee5bab977080c5d07e9acd6b7d7a4b74189c782d29587e79746201ae" Sep 11 14:45:16.659997 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.659964 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb8d8355ee5bab977080c5d07e9acd6b7d7a4b74189c782d29587e79746201ae"} err="failed to get container status \"eb8d8355ee5bab977080c5d07e9acd6b7d7a4b74189c782d29587e79746201ae\": rpc error: code = NotFound desc = could not find container \"eb8d8355ee5bab977080c5d07e9acd6b7d7a4b74189c782d29587e79746201ae\": container with ID starting with eb8d8355ee5bab977080c5d07e9acd6b7d7a4b74189c782d29587e79746201ae not found: ID does not exist" Sep 11 14:45:16.668187 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.668161 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9"] Sep 11 14:45:16.678399 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:16.678376 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-fc74dd-openshift-default-87c8d5bdb-wl4h9"] Sep 11 14:45:17.235324 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:17.235293 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:17.912407 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:17.912371 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" path="/var/lib/kubelet/pods/7df4c5e2-7ec2-4073-ae5a-075cdcb95c07/volumes" Sep 11 14:45:22.089940 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.089906 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28858c-29818965-rpfjq"] Sep 11 14:45:22.091850 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.091830 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-28858c-29818965-rpfjq"] Sep 11 14:45:22.098271 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.098241 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:22.125913 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.125884 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4"] Sep 11 14:45:22.126134 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.126089 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4" podUID="c13805ad-69a7-421e-aea0-89b18b767a2d" containerName="payload-pre-processing" containerID="cri-o://33ba01b34714006173138e5e38c6a04d5150674a2c2099255a041536ddb51b12" gracePeriod=30 Sep 11 14:45:22.206729 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.206697 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:22.323739 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.323701 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:22.375720 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.375697 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4" Sep 11 14:45:22.428559 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.428533 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c13805ad-69a7-421e-aea0-89b18b767a2d-plugins-config-volume\") pod \"c13805ad-69a7-421e-aea0-89b18b767a2d\" (UID: \"c13805ad-69a7-421e-aea0-89b18b767a2d\") " Sep 11 14:45:22.428732 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.428670 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8ppn\" (UniqueName: \"kubernetes.io/projected/c13805ad-69a7-421e-aea0-89b18b767a2d-kube-api-access-d8ppn\") pod \"c13805ad-69a7-421e-aea0-89b18b767a2d\" (UID: \"c13805ad-69a7-421e-aea0-89b18b767a2d\") " Sep 11 14:45:22.429017 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.428989 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13805ad-69a7-421e-aea0-89b18b767a2d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c13805ad-69a7-421e-aea0-89b18b767a2d" (UID: "c13805ad-69a7-421e-aea0-89b18b767a2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:45:22.430887 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.430857 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13805ad-69a7-421e-aea0-89b18b767a2d-kube-api-access-d8ppn" (OuterVolumeSpecName: "kube-api-access-d8ppn") pod "c13805ad-69a7-421e-aea0-89b18b767a2d" (UID: "c13805ad-69a7-421e-aea0-89b18b767a2d"). InnerVolumeSpecName "kube-api-access-d8ppn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:45:22.439546 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.439514 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:22.529815 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.529779 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c13805ad-69a7-421e-aea0-89b18b767a2d-plugins-config-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:45:22.529815 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.529810 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8ppn\" (UniqueName: \"kubernetes.io/projected/c13805ad-69a7-421e-aea0-89b18b767a2d-kube-api-access-d8ppn\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:45:22.680786 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.680756 2553 generic.go:356] "Generic (PLEG): container finished" podID="c13805ad-69a7-421e-aea0-89b18b767a2d" containerID="33ba01b34714006173138e5e38c6a04d5150674a2c2099255a041536ddb51b12" exitCode=0 Sep 11 14:45:22.680961 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.680798 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4" Sep 11 14:45:22.680961 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.680843 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4" event={"ID":"c13805ad-69a7-421e-aea0-89b18b767a2d","Type":"ContainerDied","Data":"33ba01b34714006173138e5e38c6a04d5150674a2c2099255a041536ddb51b12"} Sep 11 14:45:22.680961 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.680898 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4" event={"ID":"c13805ad-69a7-421e-aea0-89b18b767a2d","Type":"ContainerDied","Data":"a6efb062a4337cf49615698be99487f67f18f3238393122230f3128865c5ea62"} Sep 11 14:45:22.680961 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.680928 2553 scope.go:117] "RemoveContainer" containerID="33ba01b34714006173138e5e38c6a04d5150674a2c2099255a041536ddb51b12" Sep 11 14:45:22.690043 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.690026 2553 scope.go:117] "RemoveContainer" containerID="33ba01b34714006173138e5e38c6a04d5150674a2c2099255a041536ddb51b12" Sep 11 14:45:22.690332 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:45:22.690312 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33ba01b34714006173138e5e38c6a04d5150674a2c2099255a041536ddb51b12\": container with ID starting with 33ba01b34714006173138e5e38c6a04d5150674a2c2099255a041536ddb51b12 not found: ID does not exist" containerID="33ba01b34714006173138e5e38c6a04d5150674a2c2099255a041536ddb51b12" Sep 11 14:45:22.690378 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.690343 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33ba01b34714006173138e5e38c6a04d5150674a2c2099255a041536ddb51b12"} err="failed to get container status \"33ba01b34714006173138e5e38c6a04d5150674a2c2099255a041536ddb51b12\": rpc error: code = NotFound desc = could not find container \"33ba01b34714006173138e5e38c6a04d5150674a2c2099255a041536ddb51b12\": container with ID starting with 33ba01b34714006173138e5e38c6a04d5150674a2c2099255a041536ddb51b12 not found: ID does not exist" Sep 11 14:45:22.709265 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.709237 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4"] Sep 11 14:45:22.717434 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.717394 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-28858c-555f84bb-xqhk4"] Sep 11 14:45:22.795533 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.795501 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:22.892598 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:22.892565 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:23.751099 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:23.751062 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:23.912461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:23.912428 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ac00535-28d3-48d3-99a1-7571db396d08" path="/var/lib/kubelet/pods/5ac00535-28d3-48d3-99a1-7571db396d08/volumes" Sep 11 14:45:23.912818 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:23.912805 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c13805ad-69a7-421e-aea0-89b18b767a2d" path="/var/lib/kubelet/pods/c13805ad-69a7-421e-aea0-89b18b767a2d/volumes" Sep 11 14:45:24.677407 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:24.677356 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:25.171853 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:25.171813 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:25.613678 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:25.613594 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:29.481609 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:29.481556 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:29.643613 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:29.643579 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:29.802890 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:29.802807 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:30.387789 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:30.387755 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:31.327059 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:31.327021 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:32.230561 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:32.230524 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:33.197148 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:33.197104 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:34.136707 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:34.136669 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:40.911777 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:40.911746 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:40.969512 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:40.969478 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q"] Sep 11 14:45:40.969793 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:40.969741 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q" podUID="cdbd0164-2589-4690-8fa3-0aeb4d5a506a" containerName="payload-pre-processing" containerID="cri-o://8a33034967fbe5dbf8c525a4e751239c577afc6997ae65039b22389bd69502f8" gracePeriod=30 Sep 11 14:45:41.008347 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:41.008313 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:41.169048 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:41.168874 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:41.251362 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:41.251327 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q" Sep 11 14:45:41.283561 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:41.283530 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:41.393759 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:41.393716 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:41.414807 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:41.414771 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbd0164-2589-4690-8fa3-0aeb4d5a506a-plugins-config-volume\") pod \"cdbd0164-2589-4690-8fa3-0aeb4d5a506a\" (UID: \"cdbd0164-2589-4690-8fa3-0aeb4d5a506a\") " Sep 11 14:45:41.415000 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:41.414835 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tf8b\" (UniqueName: \"kubernetes.io/projected/cdbd0164-2589-4690-8fa3-0aeb4d5a506a-kube-api-access-4tf8b\") pod \"cdbd0164-2589-4690-8fa3-0aeb4d5a506a\" (UID: \"cdbd0164-2589-4690-8fa3-0aeb4d5a506a\") " Sep 11 14:45:41.415192 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:41.415169 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdbd0164-2589-4690-8fa3-0aeb4d5a506a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cdbd0164-2589-4690-8fa3-0aeb4d5a506a" (UID: "cdbd0164-2589-4690-8fa3-0aeb4d5a506a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:45:41.417039 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:41.417012 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdbd0164-2589-4690-8fa3-0aeb4d5a506a-kube-api-access-4tf8b" (OuterVolumeSpecName: "kube-api-access-4tf8b") pod "cdbd0164-2589-4690-8fa3-0aeb4d5a506a" (UID: "cdbd0164-2589-4690-8fa3-0aeb4d5a506a"). InnerVolumeSpecName "kube-api-access-4tf8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:45:41.516390 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:41.516356 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdbd0164-2589-4690-8fa3-0aeb4d5a506a-plugins-config-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:45:41.516390 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:41.516383 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tf8b\" (UniqueName: \"kubernetes.io/projected/cdbd0164-2589-4690-8fa3-0aeb4d5a506a-kube-api-access-4tf8b\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:45:41.783065 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:41.782979 2553 generic.go:356] "Generic (PLEG): container finished" podID="cdbd0164-2589-4690-8fa3-0aeb4d5a506a" containerID="8a33034967fbe5dbf8c525a4e751239c577afc6997ae65039b22389bd69502f8" exitCode=0 Sep 11 14:45:41.783065 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:41.783026 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q" Sep 11 14:45:41.783253 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:41.783065 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q" event={"ID":"cdbd0164-2589-4690-8fa3-0aeb4d5a506a","Type":"ContainerDied","Data":"8a33034967fbe5dbf8c525a4e751239c577afc6997ae65039b22389bd69502f8"} Sep 11 14:45:41.783253 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:41.783154 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q" event={"ID":"cdbd0164-2589-4690-8fa3-0aeb4d5a506a","Type":"ContainerDied","Data":"466e3f4a820fb0539d5117a03b086f8c6ddfac4f86b0b3d546221e104176e293"} Sep 11 14:45:41.783253 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:41.783182 2553 scope.go:117] "RemoveContainer" containerID="8a33034967fbe5dbf8c525a4e751239c577afc6997ae65039b22389bd69502f8" Sep 11 14:45:41.792282 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:41.792263 2553 scope.go:117] "RemoveContainer" containerID="8a33034967fbe5dbf8c525a4e751239c577afc6997ae65039b22389bd69502f8" Sep 11 14:45:41.792607 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:45:41.792588 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a33034967fbe5dbf8c525a4e751239c577afc6997ae65039b22389bd69502f8\": container with ID starting with 8a33034967fbe5dbf8c525a4e751239c577afc6997ae65039b22389bd69502f8 not found: ID does not exist" containerID="8a33034967fbe5dbf8c525a4e751239c577afc6997ae65039b22389bd69502f8" Sep 11 14:45:41.792702 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:41.792614 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a33034967fbe5dbf8c525a4e751239c577afc6997ae65039b22389bd69502f8"} err="failed to get container status \"8a33034967fbe5dbf8c525a4e751239c577afc6997ae65039b22389bd69502f8\": rpc error: code = NotFound desc = could not find container \"8a33034967fbe5dbf8c525a4e751239c577afc6997ae65039b22389bd69502f8\": container with ID starting with 8a33034967fbe5dbf8c525a4e751239c577afc6997ae65039b22389bd69502f8 not found: ID does not exist" Sep 11 14:45:41.823489 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:41.823457 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q"] Sep 11 14:45:41.831719 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:41.831688 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7eeb45-65d48958f8-jcg4q"] Sep 11 14:45:41.911385 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:41.911350 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:41.912501 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:41.912479 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdbd0164-2589-4690-8fa3-0aeb4d5a506a" path="/var/lib/kubelet/pods/cdbd0164-2589-4690-8fa3-0aeb4d5a506a/volumes" Sep 11 14:45:42.049312 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:42.049200 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:43.029180 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:43.029128 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:44.109563 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:44.109531 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:48.328116 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:48.328077 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:48.456864 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:48.456827 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:48.631754 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:48.631672 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm"] Sep 11 14:45:48.631913 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:48.631890 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" podUID="032871f2-166d-4a19-a966-b76fabfff473" containerName="istio-proxy" containerID="cri-o://7710cfd01c2169c3f6082a1d011b914081a7d6dd7c3126fc5360d9773371db30" gracePeriod=30 Sep 11 14:45:48.670903 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:48.670869 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:49.053117 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:49.053083 2553 patch_prober.go:28] interesting pod/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:45:49.053300 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:49.053150 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" podUID="032871f2-166d-4a19-a966-b76fabfff473" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:45:53.849409 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:53.849369 2553 generic.go:356] "Generic (PLEG): container finished" podID="032871f2-166d-4a19-a966-b76fabfff473" containerID="7710cfd01c2169c3f6082a1d011b914081a7d6dd7c3126fc5360d9773371db30" exitCode=0 Sep 11 14:45:53.849798 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:53.849450 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" event={"ID":"032871f2-166d-4a19-a966-b76fabfff473","Type":"ContainerDied","Data":"7710cfd01c2169c3f6082a1d011b914081a7d6dd7c3126fc5360d9773371db30"} Sep 11 14:45:53.849798 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:53.849515 2553 scope.go:117] "RemoveContainer" containerID="191ce6525130070439b47e91724682dfea46cf34eb34e9f42375d950d5a28890" Sep 11 14:45:53.893081 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:53.893024 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:45:54.028292 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.028257 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-credential-socket\") pod \"032871f2-166d-4a19-a966-b76fabfff473\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " Sep 11 14:45:54.028492 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.028310 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-workload-socket\") pod \"032871f2-166d-4a19-a966-b76fabfff473\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " Sep 11 14:45:54.028492 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.028347 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-workload-certs\") pod \"032871f2-166d-4a19-a966-b76fabfff473\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " Sep 11 14:45:54.028492 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.028382 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032871f2-166d-4a19-a966-b76fabfff473-istiod-ca-cert\") pod \"032871f2-166d-4a19-a966-b76fabfff473\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " Sep 11 14:45:54.028492 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.028455 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032871f2-166d-4a19-a966-b76fabfff473-istio-podinfo\") pod \"032871f2-166d-4a19-a966-b76fabfff473\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " Sep 11 14:45:54.028492 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.028489 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-istio-envoy\") pod \"032871f2-166d-4a19-a966-b76fabfff473\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " Sep 11 14:45:54.028769 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.028545 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvrx6\" (UniqueName: \"kubernetes.io/projected/032871f2-166d-4a19-a966-b76fabfff473-kube-api-access-xvrx6\") pod \"032871f2-166d-4a19-a966-b76fabfff473\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " Sep 11 14:45:54.028769 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.028597 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032871f2-166d-4a19-a966-b76fabfff473-istio-token\") pod \"032871f2-166d-4a19-a966-b76fabfff473\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " Sep 11 14:45:54.028769 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.028597 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "032871f2-166d-4a19-a966-b76fabfff473" (UID: "032871f2-166d-4a19-a966-b76fabfff473"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:45:54.028769 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.028623 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-istio-data\") pod \"032871f2-166d-4a19-a966-b76fabfff473\" (UID: \"032871f2-166d-4a19-a966-b76fabfff473\") " Sep 11 14:45:54.028769 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.028697 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "032871f2-166d-4a19-a966-b76fabfff473" (UID: "032871f2-166d-4a19-a966-b76fabfff473"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:45:54.028769 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.028715 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "032871f2-166d-4a19-a966-b76fabfff473" (UID: "032871f2-166d-4a19-a966-b76fabfff473"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:45:54.029092 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.028939 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-istio-data" (OuterVolumeSpecName: "istio-data") pod "032871f2-166d-4a19-a966-b76fabfff473" (UID: "032871f2-166d-4a19-a966-b76fabfff473"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:45:54.029092 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.028961 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/032871f2-166d-4a19-a966-b76fabfff473-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "032871f2-166d-4a19-a966-b76fabfff473" (UID: "032871f2-166d-4a19-a966-b76fabfff473"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:45:54.029092 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.029032 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-istio-data\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:45:54.029092 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.029052 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-credential-socket\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:45:54.029092 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.029062 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-workload-socket\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:45:54.029092 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.029070 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-workload-certs\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:45:54.029092 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.029080 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032871f2-166d-4a19-a966-b76fabfff473-istiod-ca-cert\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:45:54.030843 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.030815 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032871f2-166d-4a19-a966-b76fabfff473-kube-api-access-xvrx6" (OuterVolumeSpecName: "kube-api-access-xvrx6") pod "032871f2-166d-4a19-a966-b76fabfff473" (UID: "032871f2-166d-4a19-a966-b76fabfff473"). InnerVolumeSpecName "kube-api-access-xvrx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:45:54.030940 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.030878 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032871f2-166d-4a19-a966-b76fabfff473-istio-token" (OuterVolumeSpecName: "istio-token") pod "032871f2-166d-4a19-a966-b76fabfff473" (UID: "032871f2-166d-4a19-a966-b76fabfff473"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:45:54.031103 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.031079 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "032871f2-166d-4a19-a966-b76fabfff473" (UID: "032871f2-166d-4a19-a966-b76fabfff473"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:45:54.031203 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.031084 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/032871f2-166d-4a19-a966-b76fabfff473-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "032871f2-166d-4a19-a966-b76fabfff473" (UID: "032871f2-166d-4a19-a966-b76fabfff473"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:45:54.130254 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.130214 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032871f2-166d-4a19-a966-b76fabfff473-istio-token\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:45:54.130254 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.130244 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032871f2-166d-4a19-a966-b76fabfff473-istio-podinfo\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:45:54.130254 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.130253 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032871f2-166d-4a19-a966-b76fabfff473-istio-envoy\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:45:54.130254 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.130262 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvrx6\" (UniqueName: \"kubernetes.io/projected/032871f2-166d-4a19-a966-b76fabfff473-kube-api-access-xvrx6\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:45:54.857148 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.857110 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" event={"ID":"032871f2-166d-4a19-a966-b76fabfff473","Type":"ContainerDied","Data":"be07a0cb64d29a67d922a5e1b7cd6e45c8f05b919834e83750592fe2cb678b54"} Sep 11 14:45:54.857651 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.857172 2553 scope.go:117] "RemoveContainer" containerID="7710cfd01c2169c3f6082a1d011b914081a7d6dd7c3126fc5360d9773371db30" Sep 11 14:45:54.857651 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.857121 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm" Sep 11 14:45:54.886332 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.886304 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm"] Sep 11 14:45:54.890327 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:54.890303 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-7eeb45-openshift-default-7d7d7687f6-6mptm"] Sep 11 14:45:55.912230 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:55.912195 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="032871f2-166d-4a19-a966-b76fabfff473" path="/var/lib/kubelet/pods/032871f2-166d-4a19-a966-b76fabfff473/volumes" Sep 11 14:45:56.021776 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:56.021739 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:58.021847 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:58.021815 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh"] Sep 11 14:45:58.036724 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:58.036689 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:59.289581 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:59.289547 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:45:59.766282 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:45:59.766223 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:00.199494 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:00.199441 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:00.668980 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:00.668937 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:02.242652 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:02.242619 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:03.131373 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:03.131339 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:03.297913 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:03.297868 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:03.821096 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:03.821058 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:05.670513 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:05.670476 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:07.019570 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.019532 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:07.117212 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.117168 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:07.281827 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.281713 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:07.712098 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.712055 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:07.807815 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.807785 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:07.926773 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.926742 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8"] Sep 11 14:46:07.927109 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927097 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13805ad-69a7-421e-aea0-89b18b767a2d" containerName="payload-pre-processing" Sep 11 14:46:07.927154 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927111 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13805ad-69a7-421e-aea0-89b18b767a2d" containerName="payload-pre-processing" Sep 11 14:46:07.927154 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927119 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdbd0164-2589-4690-8fa3-0aeb4d5a506a" containerName="payload-pre-processing" Sep 11 14:46:07.927154 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927125 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdbd0164-2589-4690-8fa3-0aeb4d5a506a" containerName="payload-pre-processing" Sep 11 14:46:07.927154 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927133 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" containerName="istio-proxy" Sep 11 14:46:07.927154 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927138 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" containerName="istio-proxy" Sep 11 14:46:07.927154 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927147 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="032871f2-166d-4a19-a966-b76fabfff473" containerName="istio-proxy" Sep 11 14:46:07.927154 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927152 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="032871f2-166d-4a19-a966-b76fabfff473" containerName="istio-proxy" Sep 11 14:46:07.927364 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927161 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74a47a90-8e1d-4d0f-9aa9-2fb80b977711" containerName="cleanup" Sep 11 14:46:07.927364 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927166 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="74a47a90-8e1d-4d0f-9aa9-2fb80b977711" containerName="cleanup" Sep 11 14:46:07.927364 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927178 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ac00535-28d3-48d3-99a1-7571db396d08" containerName="cleanup" Sep 11 14:46:07.927364 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927183 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ac00535-28d3-48d3-99a1-7571db396d08" containerName="cleanup" Sep 11 14:46:07.927364 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927193 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" containerName="istio-proxy" Sep 11 14:46:07.927364 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927199 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" containerName="istio-proxy" Sep 11 14:46:07.927364 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927205 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd45bc23-fd9d-4dd0-bb7f-d824fee14fbe" containerName="cleanup" Sep 11 14:46:07.927364 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927209 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd45bc23-fd9d-4dd0-bb7f-d824fee14fbe" containerName="cleanup" Sep 11 14:46:07.927364 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927265 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c13805ad-69a7-421e-aea0-89b18b767a2d" containerName="payload-pre-processing" Sep 11 14:46:07.927364 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927275 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="032871f2-166d-4a19-a966-b76fabfff473" containerName="istio-proxy" Sep 11 14:46:07.927364 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927282 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="74a47a90-8e1d-4d0f-9aa9-2fb80b977711" containerName="cleanup" Sep 11 14:46:07.927364 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927286 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ac00535-28d3-48d3-99a1-7571db396d08" containerName="cleanup" Sep 11 14:46:07.927364 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927292 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdbd0164-2589-4690-8fa3-0aeb4d5a506a" containerName="payload-pre-processing" Sep 11 14:46:07.927364 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927298 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" containerName="istio-proxy" Sep 11 14:46:07.927364 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927303 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="032871f2-166d-4a19-a966-b76fabfff473" containerName="istio-proxy" Sep 11 14:46:07.927364 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927308 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd45bc23-fd9d-4dd0-bb7f-d824fee14fbe" containerName="cleanup" Sep 11 14:46:07.927364 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.927313 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7df4c5e2-7ec2-4073-ae5a-075cdcb95c07" containerName="istio-proxy" Sep 11 14:46:07.931969 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.931953 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8" Sep 11 14:46:07.936451 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.936431 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9fedd5\"" Sep 11 14:46:07.936607 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.936590 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9fedd5-dockercfg-xvl5m\"" Sep 11 14:46:07.942082 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:07.942062 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8"] Sep 11 14:46:08.068918 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:08.068831 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5616fa-696a-43b8-a294-cbcbdba5eb46-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8\" (UID: \"ec5616fa-696a-43b8-a294-cbcbdba5eb46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8" Sep 11 14:46:08.068918 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:08.068881 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m26k7\" (UniqueName: \"kubernetes.io/projected/ec5616fa-696a-43b8-a294-cbcbdba5eb46-kube-api-access-m26k7\") pod \"payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8\" (UID: \"ec5616fa-696a-43b8-a294-cbcbdba5eb46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8" Sep 11 14:46:08.169948 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:08.169914 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5616fa-696a-43b8-a294-cbcbdba5eb46-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8\" (UID: \"ec5616fa-696a-43b8-a294-cbcbdba5eb46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8" Sep 11 14:46:08.170121 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:08.169967 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m26k7\" (UniqueName: \"kubernetes.io/projected/ec5616fa-696a-43b8-a294-cbcbdba5eb46-kube-api-access-m26k7\") pod \"payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8\" (UID: \"ec5616fa-696a-43b8-a294-cbcbdba5eb46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8" Sep 11 14:46:08.170661 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:08.170643 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5616fa-696a-43b8-a294-cbcbdba5eb46-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8\" (UID: \"ec5616fa-696a-43b8-a294-cbcbdba5eb46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8" Sep 11 14:46:08.181086 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:08.181058 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m26k7\" (UniqueName: \"kubernetes.io/projected/ec5616fa-696a-43b8-a294-cbcbdba5eb46-kube-api-access-m26k7\") pod \"payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8\" (UID: \"ec5616fa-696a-43b8-a294-cbcbdba5eb46\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8" Sep 11 14:46:08.244270 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:08.244231 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8" Sep 11 14:46:08.524155 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:08.524131 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8"] Sep 11 14:46:08.526660 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:46:08.526632 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5616fa_696a_43b8_a294_cbcbdba5eb46.slice/crio-119d10a21fa683f41d7bca6557f0e442a43480f0a9909ae01694feacf32f491e WatchSource:0}: Error finding container 119d10a21fa683f41d7bca6557f0e442a43480f0a9909ae01694feacf32f491e: Status 404 returned error can't find the container with id 119d10a21fa683f41d7bca6557f0e442a43480f0a9909ae01694feacf32f491e Sep 11 14:46:08.930065 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:08.930030 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8" event={"ID":"ec5616fa-696a-43b8-a294-cbcbdba5eb46","Type":"ContainerStarted","Data":"531ee3db58955f71dd64fb9b5cb8b1fc657732f0e83a96da6b495b162f9c2450"} Sep 11 14:46:08.930253 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:08.930084 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8" event={"ID":"ec5616fa-696a-43b8-a294-cbcbdba5eb46","Type":"ContainerStarted","Data":"119d10a21fa683f41d7bca6557f0e442a43480f0a9909ae01694feacf32f491e"} Sep 11 14:46:08.930253 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:08.930148 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8" Sep 11 14:46:08.954175 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:08.954118 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8" podStartSLOduration=1.954103792 podStartE2EDuration="1.954103792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:46:08.952013418 +0000 UTC m=+1879.566119785" watchObservedRunningTime="2026-09-11 14:46:08.954103792 +0000 UTC m=+1879.568210135" Sep 11 14:46:13.041021 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.040927 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" podUID="c42f0609-3b37-4823-a21e-854e2d0ed6ac" containerName="main" containerID="cri-o://47cfc3951391027fafe4ec473b8098009d7bee0caf1f297258a6da5d8d9f7a34" gracePeriod=45 Sep 11 14:46:13.300999 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.300935 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:46:13.419579 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.419545 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-tmp-dir\") pod \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " Sep 11 14:46:13.419772 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.419603 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcd7w\" (UniqueName: \"kubernetes.io/projected/c42f0609-3b37-4823-a21e-854e2d0ed6ac-kube-api-access-bcd7w\") pod \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " Sep 11 14:46:13.419772 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.419641 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-home\") pod \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " Sep 11 14:46:13.419772 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.419663 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-dshm\") pod \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " Sep 11 14:46:13.419772 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.419692 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-model-cache\") pod \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " Sep 11 14:46:13.419772 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.419716 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42f0609-3b37-4823-a21e-854e2d0ed6ac-tls-certs\") pod \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\" (UID: \"c42f0609-3b37-4823-a21e-854e2d0ed6ac\") " Sep 11 14:46:13.420047 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.419856 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c42f0609-3b37-4823-a21e-854e2d0ed6ac" (UID: "c42f0609-3b37-4823-a21e-854e2d0ed6ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:46:13.420047 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.419921 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-home" (OuterVolumeSpecName: "home") pod "c42f0609-3b37-4823-a21e-854e2d0ed6ac" (UID: "c42f0609-3b37-4823-a21e-854e2d0ed6ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:46:13.420047 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.419960 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "c42f0609-3b37-4823-a21e-854e2d0ed6ac" (UID: "c42f0609-3b37-4823-a21e-854e2d0ed6ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:46:13.420199 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.420104 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-home\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:46:13.420199 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.420118 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-model-cache\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:46:13.420199 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.420128 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-tmp-dir\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:46:13.422030 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.422006 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42f0609-3b37-4823-a21e-854e2d0ed6ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c42f0609-3b37-4823-a21e-854e2d0ed6ac" (UID: "c42f0609-3b37-4823-a21e-854e2d0ed6ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:46:13.422030 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.422018 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-dshm" (OuterVolumeSpecName: "dshm") pod "c42f0609-3b37-4823-a21e-854e2d0ed6ac" (UID: "c42f0609-3b37-4823-a21e-854e2d0ed6ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:46:13.422030 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.422021 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42f0609-3b37-4823-a21e-854e2d0ed6ac-kube-api-access-bcd7w" (OuterVolumeSpecName: "kube-api-access-bcd7w") pod "c42f0609-3b37-4823-a21e-854e2d0ed6ac" (UID: "c42f0609-3b37-4823-a21e-854e2d0ed6ac"). InnerVolumeSpecName "kube-api-access-bcd7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:46:13.520967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.520936 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcd7w\" (UniqueName: \"kubernetes.io/projected/c42f0609-3b37-4823-a21e-854e2d0ed6ac-kube-api-access-bcd7w\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:46:13.520967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.520961 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c42f0609-3b37-4823-a21e-854e2d0ed6ac-dshm\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:46:13.520967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.520970 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c42f0609-3b37-4823-a21e-854e2d0ed6ac-tls-certs\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:46:13.957916 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.957886 2553 generic.go:356] "Generic (PLEG): container finished" podID="c42f0609-3b37-4823-a21e-854e2d0ed6ac" containerID="47cfc3951391027fafe4ec473b8098009d7bee0caf1f297258a6da5d8d9f7a34" exitCode=0 Sep 11 14:46:13.958078 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.957926 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" event={"ID":"c42f0609-3b37-4823-a21e-854e2d0ed6ac","Type":"ContainerDied","Data":"47cfc3951391027fafe4ec473b8098009d7bee0caf1f297258a6da5d8d9f7a34"} Sep 11 14:46:13.958078 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.957930 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" Sep 11 14:46:13.958078 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.957952 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh" event={"ID":"c42f0609-3b37-4823-a21e-854e2d0ed6ac","Type":"ContainerDied","Data":"1acbe24dc35b6fae86aafd088b24600f6cb9aa4bcf34ad9405a1282552d6d1e0"} Sep 11 14:46:13.958078 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.957969 2553 scope.go:117] "RemoveContainer" containerID="47cfc3951391027fafe4ec473b8098009d7bee0caf1f297258a6da5d8d9f7a34" Sep 11 14:46:13.966664 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.966645 2553 scope.go:117] "RemoveContainer" containerID="47cfc3951391027fafe4ec473b8098009d7bee0caf1f297258a6da5d8d9f7a34" Sep 11 14:46:13.966987 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:46:13.966963 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47cfc3951391027fafe4ec473b8098009d7bee0caf1f297258a6da5d8d9f7a34\": container with ID starting with 47cfc3951391027fafe4ec473b8098009d7bee0caf1f297258a6da5d8d9f7a34 not found: ID does not exist" containerID="47cfc3951391027fafe4ec473b8098009d7bee0caf1f297258a6da5d8d9f7a34" Sep 11 14:46:13.967088 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.966994 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47cfc3951391027fafe4ec473b8098009d7bee0caf1f297258a6da5d8d9f7a34"} err="failed to get container status \"47cfc3951391027fafe4ec473b8098009d7bee0caf1f297258a6da5d8d9f7a34\": rpc error: code = NotFound desc = could not find container \"47cfc3951391027fafe4ec473b8098009d7bee0caf1f297258a6da5d8d9f7a34\": container with ID starting with 47cfc3951391027fafe4ec473b8098009d7bee0caf1f297258a6da5d8d9f7a34 not found: ID does not exist" Sep 11 14:46:13.981974 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.981946 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh"] Sep 11 14:46:13.983687 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:13.983667 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-8feb68/rate-test-model-8feb68-kserve-68895c5675-mjwmh"] Sep 11 14:46:15.912233 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:15.912196 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c42f0609-3b37-4823-a21e-854e2d0ed6ac" path="/var/lib/kubelet/pods/c42f0609-3b37-4823-a21e-854e2d0ed6ac/volumes" Sep 11 14:46:19.937542 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:19.937516 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8" Sep 11 14:46:29.571821 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:29.571783 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:30.518287 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:30.518251 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:31.300669 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:31.300636 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:32.186018 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:32.185985 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:33.145273 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:33.145238 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:33.870337 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:33.870302 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:34.190944 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.190911 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6"] Sep 11 14:46:34.191527 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.191498 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c42f0609-3b37-4823-a21e-854e2d0ed6ac" containerName="main" Sep 11 14:46:34.191527 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.191519 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c42f0609-3b37-4823-a21e-854e2d0ed6ac" containerName="main" Sep 11 14:46:34.191688 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.191539 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="032871f2-166d-4a19-a966-b76fabfff473" containerName="istio-proxy" Sep 11 14:46:34.191688 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.191548 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="032871f2-166d-4a19-a966-b76fabfff473" containerName="istio-proxy" Sep 11 14:46:34.191688 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.191674 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c42f0609-3b37-4823-a21e-854e2d0ed6ac" containerName="main" Sep 11 14:46:34.194679 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.194657 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.197683 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.197662 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-92ca78-openshift-default-dockercfg-tzxvk\"" Sep 11 14:46:34.214450 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.214425 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6"] Sep 11 14:46:34.216903 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.216877 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-token\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.217029 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.216936 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe123380-908c-44fd-bdd5-0c6b14972a1a-istiod-ca-cert\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.217029 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.216967 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-podinfo\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.217029 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.217016 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-workload-certs\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.217141 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.217053 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s7zt\" (UniqueName: \"kubernetes.io/projected/fe123380-908c-44fd-bdd5-0c6b14972a1a-kube-api-access-5s7zt\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.217180 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.217133 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-workload-socket\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.217215 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.217177 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-envoy\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.217266 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.217251 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-data\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.217308 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.217295 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-credential-socket\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.234550 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.234525 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:34.317851 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.317815 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-token\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.318058 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.317864 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe123380-908c-44fd-bdd5-0c6b14972a1a-istiod-ca-cert\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.318058 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.317882 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-podinfo\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.318058 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.318001 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-workload-certs\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.318243 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.318069 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s7zt\" (UniqueName: \"kubernetes.io/projected/fe123380-908c-44fd-bdd5-0c6b14972a1a-kube-api-access-5s7zt\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.318243 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.318121 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-workload-socket\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.318243 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.318153 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-envoy\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.318243 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.318189 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-data\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.318243 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.318240 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-credential-socket\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.318553 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.318513 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-workload-socket\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.318717 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.318691 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-data\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.318800 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.318723 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-credential-socket\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.318800 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.318760 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe123380-908c-44fd-bdd5-0c6b14972a1a-istiod-ca-cert\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.318800 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.318786 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-workload-certs\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.320335 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.320312 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-envoy\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.320687 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.320665 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-podinfo\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.327548 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.327524 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-token\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.327651 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.327630 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s7zt\" (UniqueName: \"kubernetes.io/projected/fe123380-908c-44fd-bdd5-0c6b14972a1a-kube-api-access-5s7zt\") pod \"e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.436769 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.436733 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:34.505634 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.505548 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:34.793023 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.792943 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6"] Sep 11 14:46:34.796631 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:46:34.796597 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe123380_908c_44fd_bdd5_0c6b14972a1a.slice/crio-521652aa7f82c6acff638a5c1e55fa36ed576a73fea2469a8e127c53a640d1fc WatchSource:0}: Error finding container 521652aa7f82c6acff638a5c1e55fa36ed576a73fea2469a8e127c53a640d1fc: Status 404 returned error can't find the container with id 521652aa7f82c6acff638a5c1e55fa36ed576a73fea2469a8e127c53a640d1fc Sep 11 14:46:34.798918 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.798886 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:46:34.798984 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.798956 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:46:34.799027 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:34.798989 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:46:35.069801 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:35.069716 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" event={"ID":"fe123380-908c-44fd-bdd5-0c6b14972a1a","Type":"ContainerStarted","Data":"72502be9a5cbb58449d5ef51e580e0542291e6584b3b476c1845816a324be213"} Sep 11 14:46:35.069801 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:35.069767 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" event={"ID":"fe123380-908c-44fd-bdd5-0c6b14972a1a","Type":"ContainerStarted","Data":"521652aa7f82c6acff638a5c1e55fa36ed576a73fea2469a8e127c53a640d1fc"} Sep 11 14:46:35.105647 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:35.105589 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" podStartSLOduration=1.105566416 podStartE2EDuration="1.105566416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:46:35.103446463 +0000 UTC m=+1905.717552801" watchObservedRunningTime="2026-09-11 14:46:35.105566416 +0000 UTC m=+1905.719672761" Sep 11 14:46:35.506029 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:35.505986 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:35.507387 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:35.507359 2553 patch_prober.go:28] interesting pod/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Sep 11 14:46:35.507554 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:35.507447 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" podUID="fe123380-908c-44fd-bdd5-0c6b14972a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Sep 11 14:46:36.509803 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:36.509772 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:36.528440 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:36.528393 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:37.079554 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:37.079520 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:37.080498 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:37.080478 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:37.270575 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:37.270534 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:37.378753 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:37.378661 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:37.682230 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:37.682199 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:38.927980 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:38.927931 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:39.704522 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:39.704486 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:40.618266 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:40.618229 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:40.720054 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:40.720012 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:40.879742 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:40.879655 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:41.184067 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:41.184023 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:41.307132 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:41.307087 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:46:41.453393 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:41.453269 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg"] Sep 11 14:46:41.457202 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:41.457176 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg" Sep 11 14:46:41.460307 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:41.460279 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-92ca78\"" Sep 11 14:46:41.460787 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:41.460772 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-92ca78-dockercfg-szdqn\"" Sep 11 14:46:41.469765 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:41.469743 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg"] Sep 11 14:46:41.486773 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:41.486745 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed0f67bc-254a-4c16-bd37-d7b96c926784-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg\" (UID: \"ed0f67bc-254a-4c16-bd37-d7b96c926784\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg" Sep 11 14:46:41.486892 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:41.486794 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xpzp\" (UniqueName: \"kubernetes.io/projected/ed0f67bc-254a-4c16-bd37-d7b96c926784-kube-api-access-6xpzp\") pod \"payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg\" (UID: \"ed0f67bc-254a-4c16-bd37-d7b96c926784\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg" Sep 11 14:46:41.587941 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:41.587913 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed0f67bc-254a-4c16-bd37-d7b96c926784-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg\" (UID: \"ed0f67bc-254a-4c16-bd37-d7b96c926784\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg" Sep 11 14:46:41.588129 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:41.587963 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xpzp\" (UniqueName: \"kubernetes.io/projected/ed0f67bc-254a-4c16-bd37-d7b96c926784-kube-api-access-6xpzp\") pod \"payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg\" (UID: \"ed0f67bc-254a-4c16-bd37-d7b96c926784\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg" Sep 11 14:46:41.588722 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:41.588703 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed0f67bc-254a-4c16-bd37-d7b96c926784-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg\" (UID: \"ed0f67bc-254a-4c16-bd37-d7b96c926784\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg" Sep 11 14:46:41.597676 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:41.597646 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xpzp\" (UniqueName: \"kubernetes.io/projected/ed0f67bc-254a-4c16-bd37-d7b96c926784-kube-api-access-6xpzp\") pod \"payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg\" (UID: \"ed0f67bc-254a-4c16-bd37-d7b96c926784\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg" Sep 11 14:46:41.768982 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:41.768892 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg" Sep 11 14:46:42.046985 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:42.046958 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg"] Sep 11 14:46:42.049017 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:46:42.048979 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded0f67bc_254a_4c16_bd37_d7b96c926784.slice/crio-3cfb951e3cc9cee41062bda360429e7a2e81be5ef19a4311780d9482b2f47f89 WatchSource:0}: Error finding container 3cfb951e3cc9cee41062bda360429e7a2e81be5ef19a4311780d9482b2f47f89: Status 404 returned error can't find the container with id 3cfb951e3cc9cee41062bda360429e7a2e81be5ef19a4311780d9482b2f47f89 Sep 11 14:46:42.106225 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:42.106200 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg" event={"ID":"ed0f67bc-254a-4c16-bd37-d7b96c926784","Type":"ContainerStarted","Data":"3cfb951e3cc9cee41062bda360429e7a2e81be5ef19a4311780d9482b2f47f89"} Sep 11 14:46:43.112927 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:43.112888 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg" event={"ID":"ed0f67bc-254a-4c16-bd37-d7b96c926784","Type":"ContainerStarted","Data":"13ce19a2cdf891f648f84808f7affdf458390b519533f98ccece5e4a28c07cc6"} Sep 11 14:46:43.113346 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:43.113031 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg" Sep 11 14:46:43.135013 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:43.134968 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg" podStartSLOduration=2.134951141 podStartE2EDuration="2.134951141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:46:43.133844011 +0000 UTC m=+1913.747950366" watchObservedRunningTime="2026-09-11 14:46:43.134951141 +0000 UTC m=+1913.749057519" Sep 11 14:46:53.120606 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:53.120567 2553 patch_prober.go:28] interesting pod/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 14:46:53.121042 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:53.120624 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" podUID="fe123380-908c-44fd-bdd5-0c6b14972a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 14:46:54.120593 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:54.120566 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg" Sep 11 14:46:54.181248 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:54.181218 2553 generic.go:356] "Generic (PLEG): container finished" podID="fe123380-908c-44fd-bdd5-0c6b14972a1a" containerID="72502be9a5cbb58449d5ef51e580e0542291e6584b3b476c1845816a324be213" exitCode=0 Sep 11 14:46:54.181726 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:54.181291 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" event={"ID":"fe123380-908c-44fd-bdd5-0c6b14972a1a","Type":"ContainerDied","Data":"72502be9a5cbb58449d5ef51e580e0542291e6584b3b476c1845816a324be213"} Sep 11 14:46:54.181841 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:54.181824 2553 scope.go:117] "RemoveContainer" containerID="72502be9a5cbb58449d5ef51e580e0542291e6584b3b476c1845816a324be213" Sep 11 14:46:54.183024 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:54.182999 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:46:54.183085 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:54.183070 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:46:54.183142 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:54.183127 2553 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 14:46:55.188451 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:55.188399 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" event={"ID":"fe123380-908c-44fd-bdd5-0c6b14972a1a","Type":"ContainerStarted","Data":"e3b1bf0d672445713c5b45fda0aac395985a1625b7311abce94b30f9c268c135"} Sep 11 14:46:55.506658 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:55.506565 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:46:55.506855 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:55.506833 2553 patch_prober.go:28] interesting pod/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Sep 11 14:46:55.506903 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:55.506882 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" podUID="fe123380-908c-44fd-bdd5-0c6b14972a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Sep 11 14:46:56.505955 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:56.505917 2553 patch_prober.go:28] interesting pod/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Sep 11 14:46:56.506359 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:56.505972 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" podUID="fe123380-908c-44fd-bdd5-0c6b14972a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Sep 11 14:46:57.506365 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:57.506329 2553 patch_prober.go:28] interesting pod/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Sep 11 14:46:57.506758 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:57.506385 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" podUID="fe123380-908c-44fd-bdd5-0c6b14972a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Sep 11 14:46:58.506313 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:58.506274 2553 patch_prober.go:28] interesting pod/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Sep 11 14:46:58.506504 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:58.506336 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" podUID="fe123380-908c-44fd-bdd5-0c6b14972a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Sep 11 14:46:59.506407 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:59.506377 2553 patch_prober.go:28] interesting pod/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Sep 11 14:46:59.506407 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:46:59.506445 2553 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" podUID="fe123380-908c-44fd-bdd5-0c6b14972a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Sep 11 14:47:00.510761 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:47:00.510724 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:47:00.511188 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:47:00.511129 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:47:00.511880 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:47:00.511859 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:47:02.970214 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:47:02.970177 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:04.087388 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:47:04.087354 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:04.814494 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:47:04.814460 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:05.586051 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:47:05.586020 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:06.379155 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:47:06.379123 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:07.368955 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:47:07.368917 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:12.586361 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:47:12.586322 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-cvn67_35517420-4a51-4512-810b-834bb79de2f0/payload-processing/0.log" Sep 11 14:47:12.724590 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:47:12.724556 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg_ed0f67bc-254a-4c16-bd37-d7b96c926784/payload-processing/0.log" Sep 11 14:47:20.832755 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:47:20.832714 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:33.906107 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:47:33.906066 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:55.169625 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:47:55.169537 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:58.891383 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:47:58.891351 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-cvn67_35517420-4a51-4512-810b-834bb79de2f0/payload-processing/0.log" Sep 11 14:47:59.013481 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:47:59.013453 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg_ed0f67bc-254a-4c16-bd37-d7b96c926784/payload-processing/0.log" Sep 11 14:47:59.308760 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:47:59.308668 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:47:59.687480 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:47:59.687445 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:00.705821 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:00.705788 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:01.930955 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:01.930925 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:02.656510 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:02.656451 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:02.770349 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:02.770314 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:03.055054 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:03.054960 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:04.123590 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:04.123559 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:05.096476 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:05.096436 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:05.930663 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:05.930627 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:06.799528 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:06.799490 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:06.899367 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:06.899321 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:07.074323 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:07.074223 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:07.369743 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:07.369650 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:07.467410 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:07.467371 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:07.643208 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:07.643178 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k"] Sep 11 14:48:07.646926 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:07.646911 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k" Sep 11 14:48:07.651027 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:07.651006 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-55ef8c\"" Sep 11 14:48:07.662063 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:07.662044 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-55ef8c-dockercfg-24gzm\"" Sep 11 14:48:07.678563 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:07.678540 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k"] Sep 11 14:48:07.712767 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:07.712737 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxbsm\" (UniqueName: \"kubernetes.io/projected/894ceac9-3c0c-4d46-a9db-a043bfc2572c-kube-api-access-dxbsm\") pod \"payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k\" (UID: \"894ceac9-3c0c-4d46-a9db-a043bfc2572c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k" Sep 11 14:48:07.712958 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:07.712831 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/894ceac9-3c0c-4d46-a9db-a043bfc2572c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k\" (UID: \"894ceac9-3c0c-4d46-a9db-a043bfc2572c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k" Sep 11 14:48:07.813734 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:07.813698 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/894ceac9-3c0c-4d46-a9db-a043bfc2572c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k\" (UID: \"894ceac9-3c0c-4d46-a9db-a043bfc2572c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k" Sep 11 14:48:07.813940 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:07.813792 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxbsm\" (UniqueName: \"kubernetes.io/projected/894ceac9-3c0c-4d46-a9db-a043bfc2572c-kube-api-access-dxbsm\") pod \"payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k\" (UID: \"894ceac9-3c0c-4d46-a9db-a043bfc2572c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k" Sep 11 14:48:07.814385 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:07.814365 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/894ceac9-3c0c-4d46-a9db-a043bfc2572c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k\" (UID: \"894ceac9-3c0c-4d46-a9db-a043bfc2572c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k" Sep 11 14:48:07.825265 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:07.825239 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxbsm\" (UniqueName: \"kubernetes.io/projected/894ceac9-3c0c-4d46-a9db-a043bfc2572c-kube-api-access-dxbsm\") pod \"payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k\" (UID: \"894ceac9-3c0c-4d46-a9db-a043bfc2572c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k" Sep 11 14:48:07.957204 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:07.957114 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k" Sep 11 14:48:08.233514 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:08.233474 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k"] Sep 11 14:48:08.234314 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:48:08.234283 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894ceac9_3c0c_4d46_a9db_a043bfc2572c.slice/crio-c35da4d57fa0a1e56ea3e5b0cfd96fdbe50642dea0cc737a1ede8a9a956f0c93 WatchSource:0}: Error finding container c35da4d57fa0a1e56ea3e5b0cfd96fdbe50642dea0cc737a1ede8a9a956f0c93: Status 404 returned error can't find the container with id c35da4d57fa0a1e56ea3e5b0cfd96fdbe50642dea0cc737a1ede8a9a956f0c93 Sep 11 14:48:08.563802 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:08.563698 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k" event={"ID":"894ceac9-3c0c-4d46-a9db-a043bfc2572c","Type":"ContainerStarted","Data":"0f881bbdbb568b1ac13df8e19331eaa3358d8973214da17520b343cb40e39be4"} Sep 11 14:48:08.563802 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:08.563770 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k" event={"ID":"894ceac9-3c0c-4d46-a9db-a043bfc2572c","Type":"ContainerStarted","Data":"c35da4d57fa0a1e56ea3e5b0cfd96fdbe50642dea0cc737a1ede8a9a956f0c93"} Sep 11 14:48:08.564016 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:08.563816 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k" Sep 11 14:48:08.587783 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:08.587723 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k" podStartSLOduration=1.587702916 podStartE2EDuration="1.587702916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:48:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:48:08.585398976 +0000 UTC m=+1999.199505322" watchObservedRunningTime="2026-09-11 14:48:08.587702916 +0000 UTC m=+1999.201809261" Sep 11 14:48:19.570350 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:19.570318 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k" Sep 11 14:48:30.737351 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:30.737313 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:31.662848 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:31.662811 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:32.558695 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:32.558665 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:32.836136 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:32.836021 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:33.794882 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:33.794846 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:34.851317 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:34.851283 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:35.739882 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:35.739846 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:36.583864 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:36.583832 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:37.608369 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:37.608337 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:38.492824 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:38.492777 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:39.302120 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:39.302076 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:44.001959 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:44.001915 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:44.758689 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:44.758653 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:44.795865 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:44.795824 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k"] Sep 11 14:48:44.796094 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:44.796064 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k" podUID="894ceac9-3c0c-4d46-a9db-a043bfc2572c" containerName="payload-processing" containerID="cri-o://0f881bbdbb568b1ac13df8e19331eaa3358d8973214da17520b343cb40e39be4" gracePeriod=30 Sep 11 14:48:44.860426 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:44.860391 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:44.962365 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:44.962328 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:45.049077 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:45.049053 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k" Sep 11 14:48:45.059810 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:45.059778 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:45.070731 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:45.070707 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxbsm\" (UniqueName: \"kubernetes.io/projected/894ceac9-3c0c-4d46-a9db-a043bfc2572c-kube-api-access-dxbsm\") pod \"894ceac9-3c0c-4d46-a9db-a043bfc2572c\" (UID: \"894ceac9-3c0c-4d46-a9db-a043bfc2572c\") " Sep 11 14:48:45.070848 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:45.070756 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/894ceac9-3c0c-4d46-a9db-a043bfc2572c-plugins-config-volume\") pod \"894ceac9-3c0c-4d46-a9db-a043bfc2572c\" (UID: \"894ceac9-3c0c-4d46-a9db-a043bfc2572c\") " Sep 11 14:48:45.071351 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:45.071322 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/894ceac9-3c0c-4d46-a9db-a043bfc2572c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "894ceac9-3c0c-4d46-a9db-a043bfc2572c" (UID: "894ceac9-3c0c-4d46-a9db-a043bfc2572c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:48:45.073390 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:45.073350 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894ceac9-3c0c-4d46-a9db-a043bfc2572c-kube-api-access-dxbsm" (OuterVolumeSpecName: "kube-api-access-dxbsm") pod "894ceac9-3c0c-4d46-a9db-a043bfc2572c" (UID: "894ceac9-3c0c-4d46-a9db-a043bfc2572c"). InnerVolumeSpecName "kube-api-access-dxbsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:48:45.172224 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:45.172187 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxbsm\" (UniqueName: \"kubernetes.io/projected/894ceac9-3c0c-4d46-a9db-a043bfc2572c-kube-api-access-dxbsm\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:48:45.172224 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:45.172218 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/894ceac9-3c0c-4d46-a9db-a043bfc2572c-plugins-config-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:48:45.505861 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:45.505821 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:45.689018 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:45.688983 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:45.754893 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:45.754860 2553 generic.go:356] "Generic (PLEG): container finished" podID="894ceac9-3c0c-4d46-a9db-a043bfc2572c" containerID="0f881bbdbb568b1ac13df8e19331eaa3358d8973214da17520b343cb40e39be4" exitCode=0 Sep 11 14:48:45.755088 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:45.754904 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k" Sep 11 14:48:45.755088 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:45.754941 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k" event={"ID":"894ceac9-3c0c-4d46-a9db-a043bfc2572c","Type":"ContainerDied","Data":"0f881bbdbb568b1ac13df8e19331eaa3358d8973214da17520b343cb40e39be4"} Sep 11 14:48:45.755088 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:45.755000 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k" event={"ID":"894ceac9-3c0c-4d46-a9db-a043bfc2572c","Type":"ContainerDied","Data":"c35da4d57fa0a1e56ea3e5b0cfd96fdbe50642dea0cc737a1ede8a9a956f0c93"} Sep 11 14:48:45.755088 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:45.755031 2553 scope.go:117] "RemoveContainer" containerID="0f881bbdbb568b1ac13df8e19331eaa3358d8973214da17520b343cb40e39be4" Sep 11 14:48:45.764842 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:45.764826 2553 scope.go:117] "RemoveContainer" containerID="0f881bbdbb568b1ac13df8e19331eaa3358d8973214da17520b343cb40e39be4" Sep 11 14:48:45.765141 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:48:45.765122 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f881bbdbb568b1ac13df8e19331eaa3358d8973214da17520b343cb40e39be4\": container with ID starting with 0f881bbdbb568b1ac13df8e19331eaa3358d8973214da17520b343cb40e39be4 not found: ID does not exist" containerID="0f881bbdbb568b1ac13df8e19331eaa3358d8973214da17520b343cb40e39be4" Sep 11 14:48:45.765205 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:45.765149 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f881bbdbb568b1ac13df8e19331eaa3358d8973214da17520b343cb40e39be4"} err="failed to get container status \"0f881bbdbb568b1ac13df8e19331eaa3358d8973214da17520b343cb40e39be4\": rpc error: code = NotFound desc = could not find container \"0f881bbdbb568b1ac13df8e19331eaa3358d8973214da17520b343cb40e39be4\": container with ID starting with 0f881bbdbb568b1ac13df8e19331eaa3358d8973214da17520b343cb40e39be4 not found: ID does not exist" Sep 11 14:48:45.781924 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:45.781896 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k"] Sep 11 14:48:45.783761 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:45.783743 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-55ef8c-7878c55c9f-t2d6k"] Sep 11 14:48:45.912008 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:45.911970 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="894ceac9-3c0c-4d46-a9db-a043bfc2572c" path="/var/lib/kubelet/pods/894ceac9-3c0c-4d46-a9db-a043bfc2572c/volumes" Sep 11 14:48:46.265109 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:46.265066 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:47.147835 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:47.147796 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:47.980929 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:47.980895 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:48.794080 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:48.794046 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:49.964687 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:49.964653 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:51.945246 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:51.945207 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:52.065872 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:52.065841 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:52.384973 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:52.384884 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:56.211156 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:56.211119 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9fedd5-vmb69"] Sep 11 14:48:56.211587 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:56.211536 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="894ceac9-3c0c-4d46-a9db-a043bfc2572c" containerName="payload-processing" Sep 11 14:48:56.211587 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:56.211549 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="894ceac9-3c0c-4d46-a9db-a043bfc2572c" containerName="payload-processing" Sep 11 14:48:56.211668 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:56.211616 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="894ceac9-3c0c-4d46-a9db-a043bfc2572c" containerName="payload-processing" Sep 11 14:48:56.214618 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:56.214600 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9fedd5-vmb69" Sep 11 14:48:56.218066 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:56.218046 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6975v\"" Sep 11 14:48:56.218524 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:56.218502 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:48:56.230391 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:56.230366 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9fedd5-vmb69"] Sep 11 14:48:56.272783 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:56.272752 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c00c8577-0541-4e4f-8198-38cf7f5f2062-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9fedd5-vmb69\" (UID: \"c00c8577-0541-4e4f-8198-38cf7f5f2062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9fedd5-vmb69" Sep 11 14:48:56.354805 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:56.354771 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:56.373852 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:56.373815 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c00c8577-0541-4e4f-8198-38cf7f5f2062-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9fedd5-vmb69\" (UID: \"c00c8577-0541-4e4f-8198-38cf7f5f2062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9fedd5-vmb69" Sep 11 14:48:56.374454 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:56.374437 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c00c8577-0541-4e4f-8198-38cf7f5f2062-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-9fedd5-vmb69\" (UID: \"c00c8577-0541-4e4f-8198-38cf7f5f2062\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9fedd5-vmb69" Sep 11 14:48:56.525543 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:56.525459 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9fedd5-vmb69" Sep 11 14:48:56.804604 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:56.804579 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9fedd5-vmb69"] Sep 11 14:48:56.806856 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:48:56.806823 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00c8577_0541_4e4f_8198_38cf7f5f2062.slice/crio-cb9b4e25b0d49d131ea5cad11eb03b0f73163109b015d0455d302a737b496d8e WatchSource:0}: Error finding container cb9b4e25b0d49d131ea5cad11eb03b0f73163109b015d0455d302a737b496d8e: Status 404 returned error can't find the container with id cb9b4e25b0d49d131ea5cad11eb03b0f73163109b015d0455d302a737b496d8e Sep 11 14:48:56.820837 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:56.820813 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9fedd5-vmb69" event={"ID":"c00c8577-0541-4e4f-8198-38cf7f5f2062","Type":"ContainerStarted","Data":"cb9b4e25b0d49d131ea5cad11eb03b0f73163109b015d0455d302a737b496d8e"} Sep 11 14:48:57.331727 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:57.331610 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:57.828845 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:57.828811 2553 generic.go:356] "Generic (PLEG): container finished" podID="c00c8577-0541-4e4f-8198-38cf7f5f2062" containerID="dc34a169e4a2778e71252d8abb66b518f7c9e2fe87c30c9e856679c6b0a57e20" exitCode=0 Sep 11 14:48:57.829035 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:57.828905 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9fedd5-vmb69" event={"ID":"c00c8577-0541-4e4f-8198-38cf7f5f2062","Type":"ContainerDied","Data":"dc34a169e4a2778e71252d8abb66b518f7c9e2fe87c30c9e856679c6b0a57e20"} Sep 11 14:48:58.313015 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:58.312972 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:58.975513 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:58.975487 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9fedd5-vmb69" Sep 11 14:48:59.098304 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:59.098213 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c00c8577-0541-4e4f-8198-38cf7f5f2062-maas-api-service-ca\") pod \"c00c8577-0541-4e4f-8198-38cf7f5f2062\" (UID: \"c00c8577-0541-4e4f-8198-38cf7f5f2062\") " Sep 11 14:48:59.098611 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:59.098590 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00c8577-0541-4e4f-8198-38cf7f5f2062-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c00c8577-0541-4e4f-8198-38cf7f5f2062" (UID: "c00c8577-0541-4e4f-8198-38cf7f5f2062"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:48:59.199711 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:59.199683 2553 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c00c8577-0541-4e4f-8198-38cf7f5f2062-maas-api-service-ca\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:48:59.389007 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:59.388973 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:48:59.840847 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:59.840760 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9fedd5-vmb69" event={"ID":"c00c8577-0541-4e4f-8198-38cf7f5f2062","Type":"ContainerDied","Data":"cb9b4e25b0d49d131ea5cad11eb03b0f73163109b015d0455d302a737b496d8e"} Sep 11 14:48:59.840847 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:59.840781 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9fedd5-vmb69" Sep 11 14:48:59.840847 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:48:59.840806 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb9b4e25b0d49d131ea5cad11eb03b0f73163109b015d0455d302a737b496d8e" Sep 11 14:49:00.348246 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:00.348202 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:49:01.163921 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:01.163870 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:49:11.405126 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:11.405082 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:49:11.505368 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:11.505328 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:49:12.107094 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.107010 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:49:12.174155 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.174110 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8"] Sep 11 14:49:12.174468 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.174440 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8" podUID="ec5616fa-696a-43b8-a294-cbcbdba5eb46" containerName="payload-pre-processing" containerID="cri-o://531ee3db58955f71dd64fb9b5cb8b1fc657732f0e83a96da6b495b162f9c2450" gracePeriod=30 Sep 11 14:49:12.214548 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.214504 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:49:12.313541 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.313506 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:49:12.399814 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.399773 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9fedd5-vmb69"] Sep 11 14:49:12.402967 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.402920 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:49:12.403389 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.403368 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-9fedd5-vmb69"] Sep 11 14:49:12.443609 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.443583 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8" Sep 11 14:49:12.475769 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.475729 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:49:12.533705 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.533675 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5616fa-696a-43b8-a294-cbcbdba5eb46-plugins-config-volume\") pod \"ec5616fa-696a-43b8-a294-cbcbdba5eb46\" (UID: \"ec5616fa-696a-43b8-a294-cbcbdba5eb46\") " Sep 11 14:49:12.533886 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.533719 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m26k7\" (UniqueName: \"kubernetes.io/projected/ec5616fa-696a-43b8-a294-cbcbdba5eb46-kube-api-access-m26k7\") pod \"ec5616fa-696a-43b8-a294-cbcbdba5eb46\" (UID: \"ec5616fa-696a-43b8-a294-cbcbdba5eb46\") " Sep 11 14:49:12.534136 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.534108 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec5616fa-696a-43b8-a294-cbcbdba5eb46-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec5616fa-696a-43b8-a294-cbcbdba5eb46" (UID: "ec5616fa-696a-43b8-a294-cbcbdba5eb46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:49:12.535819 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.535799 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec5616fa-696a-43b8-a294-cbcbdba5eb46-kube-api-access-m26k7" (OuterVolumeSpecName: "kube-api-access-m26k7") pod "ec5616fa-696a-43b8-a294-cbcbdba5eb46" (UID: "ec5616fa-696a-43b8-a294-cbcbdba5eb46"). InnerVolumeSpecName "kube-api-access-m26k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:49:12.634532 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.634480 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec5616fa-696a-43b8-a294-cbcbdba5eb46-plugins-config-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:49:12.634532 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.634527 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m26k7\" (UniqueName: \"kubernetes.io/projected/ec5616fa-696a-43b8-a294-cbcbdba5eb46-kube-api-access-m26k7\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:49:12.808531 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.808431 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:49:12.908372 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.908337 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:49:12.911504 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.911477 2553 generic.go:356] "Generic (PLEG): container finished" podID="ec5616fa-696a-43b8-a294-cbcbdba5eb46" containerID="531ee3db58955f71dd64fb9b5cb8b1fc657732f0e83a96da6b495b162f9c2450" exitCode=0 Sep 11 14:49:12.911621 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.911522 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8" Sep 11 14:49:12.911621 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.911532 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8" event={"ID":"ec5616fa-696a-43b8-a294-cbcbdba5eb46","Type":"ContainerDied","Data":"531ee3db58955f71dd64fb9b5cb8b1fc657732f0e83a96da6b495b162f9c2450"} Sep 11 14:49:12.911621 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.911581 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8" event={"ID":"ec5616fa-696a-43b8-a294-cbcbdba5eb46","Type":"ContainerDied","Data":"119d10a21fa683f41d7bca6557f0e442a43480f0a9909ae01694feacf32f491e"} Sep 11 14:49:12.911621 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.911612 2553 scope.go:117] "RemoveContainer" containerID="531ee3db58955f71dd64fb9b5cb8b1fc657732f0e83a96da6b495b162f9c2450" Sep 11 14:49:12.921199 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.921179 2553 scope.go:117] "RemoveContainer" containerID="531ee3db58955f71dd64fb9b5cb8b1fc657732f0e83a96da6b495b162f9c2450" Sep 11 14:49:12.921504 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:49:12.921486 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"531ee3db58955f71dd64fb9b5cb8b1fc657732f0e83a96da6b495b162f9c2450\": container with ID starting with 531ee3db58955f71dd64fb9b5cb8b1fc657732f0e83a96da6b495b162f9c2450 not found: ID does not exist" containerID="531ee3db58955f71dd64fb9b5cb8b1fc657732f0e83a96da6b495b162f9c2450" Sep 11 14:49:12.921563 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.921512 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531ee3db58955f71dd64fb9b5cb8b1fc657732f0e83a96da6b495b162f9c2450"} err="failed to get container status \"531ee3db58955f71dd64fb9b5cb8b1fc657732f0e83a96da6b495b162f9c2450\": rpc error: code = NotFound desc = could not find container \"531ee3db58955f71dd64fb9b5cb8b1fc657732f0e83a96da6b495b162f9c2450\": container with ID starting with 531ee3db58955f71dd64fb9b5cb8b1fc657732f0e83a96da6b495b162f9c2450 not found: ID does not exist" Sep 11 14:49:12.938875 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.938847 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8"] Sep 11 14:49:12.940664 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:12.940644 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9fedd5-79b8cfb4bb-q7sf8"] Sep 11 14:49:13.656267 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:13.656233 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:49:13.912264 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:13.912180 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c00c8577-0541-4e4f-8198-38cf7f5f2062" path="/var/lib/kubelet/pods/c00c8577-0541-4e4f-8198-38cf7f5f2062/volumes" Sep 11 14:49:13.912606 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:13.912591 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec5616fa-696a-43b8-a294-cbcbdba5eb46" path="/var/lib/kubelet/pods/ec5616fa-696a-43b8-a294-cbcbdba5eb46/volumes" Sep 11 14:49:14.436897 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:14.436865 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:49:15.351708 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:15.351659 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:49:17.873804 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:17.873765 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:49:21.068848 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:21.068815 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:49:49.146965 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:49.146927 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:49:49.223366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:49.223331 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:49:50.035692 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:50.035664 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmqjh_6b2452d9-1730-458c-b380-dbcfd9e5be19/ovn-acl-logging/0.log" Sep 11 14:49:50.041060 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:50.041038 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmqjh_6b2452d9-1730-458c-b380-dbcfd9e5be19/ovn-acl-logging/0.log" Sep 11 14:49:54.286258 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:54.286209 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:49:55.351693 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:49:55.351658 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:01.299378 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:01.299342 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:01.471663 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:01.471622 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:02.208765 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:02.208731 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:02.478611 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:02.478522 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:03.339555 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:03.339517 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:04.222801 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:04.222764 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:05.059189 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:05.059154 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:05.976808 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:05.976774 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:12.937982 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:12.937947 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:12.959469 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:12.958838 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg"] Sep 11 14:50:12.959469 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:12.959070 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg" podUID="ed0f67bc-254a-4c16-bd37-d7b96c926784" containerName="payload-processing" containerID="cri-o://13ce19a2cdf891f648f84808f7affdf458390b519533f98ccece5e4a28c07cc6" gracePeriod=30 Sep 11 14:50:13.074309 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:13.074279 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:13.218895 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:13.218869 2553 generic.go:356] "Generic (PLEG): container finished" podID="ed0f67bc-254a-4c16-bd37-d7b96c926784" containerID="13ce19a2cdf891f648f84808f7affdf458390b519533f98ccece5e4a28c07cc6" exitCode=0 Sep 11 14:50:13.219035 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:13.218936 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg" event={"ID":"ed0f67bc-254a-4c16-bd37-d7b96c926784","Type":"ContainerDied","Data":"13ce19a2cdf891f648f84808f7affdf458390b519533f98ccece5e4a28c07cc6"} Sep 11 14:50:13.219035 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:13.218984 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg" event={"ID":"ed0f67bc-254a-4c16-bd37-d7b96c926784","Type":"ContainerDied","Data":"3cfb951e3cc9cee41062bda360429e7a2e81be5ef19a4311780d9482b2f47f89"} Sep 11 14:50:13.219035 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:13.219002 2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cfb951e3cc9cee41062bda360429e7a2e81be5ef19a4311780d9482b2f47f89" Sep 11 14:50:13.220190 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:13.220176 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg" Sep 11 14:50:13.307338 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:13.307302 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xpzp\" (UniqueName: \"kubernetes.io/projected/ed0f67bc-254a-4c16-bd37-d7b96c926784-kube-api-access-6xpzp\") pod \"ed0f67bc-254a-4c16-bd37-d7b96c926784\" (UID: \"ed0f67bc-254a-4c16-bd37-d7b96c926784\") " Sep 11 14:50:13.307569 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:13.307381 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed0f67bc-254a-4c16-bd37-d7b96c926784-plugins-config-volume\") pod \"ed0f67bc-254a-4c16-bd37-d7b96c926784\" (UID: \"ed0f67bc-254a-4c16-bd37-d7b96c926784\") " Sep 11 14:50:13.307794 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:13.307770 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed0f67bc-254a-4c16-bd37-d7b96c926784-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed0f67bc-254a-4c16-bd37-d7b96c926784" (UID: "ed0f67bc-254a-4c16-bd37-d7b96c926784"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:50:13.309494 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:13.309471 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed0f67bc-254a-4c16-bd37-d7b96c926784-kube-api-access-6xpzp" (OuterVolumeSpecName: "kube-api-access-6xpzp") pod "ed0f67bc-254a-4c16-bd37-d7b96c926784" (UID: "ed0f67bc-254a-4c16-bd37-d7b96c926784"). InnerVolumeSpecName "kube-api-access-6xpzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:50:13.310904 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:13.310879 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:13.408496 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:13.408457 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xpzp\" (UniqueName: \"kubernetes.io/projected/ed0f67bc-254a-4c16-bd37-d7b96c926784-kube-api-access-6xpzp\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:13.408496 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:13.408488 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed0f67bc-254a-4c16-bd37-d7b96c926784-plugins-config-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:13.641068 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:13.641004 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:13.739779 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:13.739742 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:14.223818 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:14.223722 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg" Sep 11 14:50:14.243970 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:14.243944 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg"] Sep 11 14:50:14.251732 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:14.251707 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-92ca78-bcb5bb6d7-684hg"] Sep 11 14:50:14.572587 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:14.572495 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:15.423520 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:15.423483 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:15.912112 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:15.912079 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed0f67bc-254a-4c16-bd37-d7b96c926784" path="/var/lib/kubelet/pods/ed0f67bc-254a-4c16-bd37-d7b96c926784/volumes" Sep 11 14:50:20.421498 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:20.421464 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:20.512344 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:20.512283 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:20.776064 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:20.775974 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6"] Sep 11 14:50:20.776245 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:20.776208 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" podUID="fe123380-908c-44fd-bdd5-0c6b14972a1a" containerName="istio-proxy" containerID="cri-o://e3b1bf0d672445713c5b45fda0aac395985a1625b7311abce94b30f9c268c135" gracePeriod=30 Sep 11 14:50:20.790361 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:20.790322 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:21.327438 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:21.327367 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:22.396319 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:22.396284 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:23.389039 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:23.389007 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:24.176780 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:24.176742 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:26.024550 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.024523 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:50:26.129766 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.129671 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-data\") pod \"fe123380-908c-44fd-bdd5-0c6b14972a1a\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " Sep 11 14:50:26.129766 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.129719 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-workload-certs\") pod \"fe123380-908c-44fd-bdd5-0c6b14972a1a\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " Sep 11 14:50:26.129766 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.129755 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5s7zt\" (UniqueName: \"kubernetes.io/projected/fe123380-908c-44fd-bdd5-0c6b14972a1a-kube-api-access-5s7zt\") pod \"fe123380-908c-44fd-bdd5-0c6b14972a1a\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " Sep 11 14:50:26.130070 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.129780 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-workload-socket\") pod \"fe123380-908c-44fd-bdd5-0c6b14972a1a\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " Sep 11 14:50:26.130070 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.129849 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-envoy\") pod \"fe123380-908c-44fd-bdd5-0c6b14972a1a\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " Sep 11 14:50:26.130070 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.129881 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe123380-908c-44fd-bdd5-0c6b14972a1a-istiod-ca-cert\") pod \"fe123380-908c-44fd-bdd5-0c6b14972a1a\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " Sep 11 14:50:26.130070 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.129925 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-token\") pod \"fe123380-908c-44fd-bdd5-0c6b14972a1a\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " Sep 11 14:50:26.130070 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.129956 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-credential-socket\") pod \"fe123380-908c-44fd-bdd5-0c6b14972a1a\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " Sep 11 14:50:26.130070 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.129991 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-podinfo\") pod \"fe123380-908c-44fd-bdd5-0c6b14972a1a\" (UID: \"fe123380-908c-44fd-bdd5-0c6b14972a1a\") " Sep 11 14:50:26.130070 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.129984 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-data" (OuterVolumeSpecName: "istio-data") pod "fe123380-908c-44fd-bdd5-0c6b14972a1a" (UID: "fe123380-908c-44fd-bdd5-0c6b14972a1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:50:26.130461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.130073 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fe123380-908c-44fd-bdd5-0c6b14972a1a" (UID: "fe123380-908c-44fd-bdd5-0c6b14972a1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:50:26.130461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.130089 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fe123380-908c-44fd-bdd5-0c6b14972a1a" (UID: "fe123380-908c-44fd-bdd5-0c6b14972a1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:50:26.130461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.130365 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-data\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:26.130461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.130369 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe123380-908c-44fd-bdd5-0c6b14972a1a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fe123380-908c-44fd-bdd5-0c6b14972a1a" (UID: "fe123380-908c-44fd-bdd5-0c6b14972a1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:50:26.130461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.130386 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-workload-certs\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:26.130461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.130399 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-workload-socket\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:26.130725 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.130527 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fe123380-908c-44fd-bdd5-0c6b14972a1a" (UID: "fe123380-908c-44fd-bdd5-0c6b14972a1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:50:26.132221 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.132190 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-token" (OuterVolumeSpecName: "istio-token") pod "fe123380-908c-44fd-bdd5-0c6b14972a1a" (UID: "fe123380-908c-44fd-bdd5-0c6b14972a1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:50:26.132300 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.132259 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fe123380-908c-44fd-bdd5-0c6b14972a1a" (UID: "fe123380-908c-44fd-bdd5-0c6b14972a1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:50:26.132597 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.132570 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe123380-908c-44fd-bdd5-0c6b14972a1a-kube-api-access-5s7zt" (OuterVolumeSpecName: "kube-api-access-5s7zt") pod "fe123380-908c-44fd-bdd5-0c6b14972a1a" (UID: "fe123380-908c-44fd-bdd5-0c6b14972a1a"). InnerVolumeSpecName "kube-api-access-5s7zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:50:26.132655 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.132580 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fe123380-908c-44fd-bdd5-0c6b14972a1a" (UID: "fe123380-908c-44fd-bdd5-0c6b14972a1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:50:26.231457 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.231396 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-envoy\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:26.231457 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.231451 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe123380-908c-44fd-bdd5-0c6b14972a1a-istiod-ca-cert\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:26.231457 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.231460 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-token\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:26.231696 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.231468 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe123380-908c-44fd-bdd5-0c6b14972a1a-credential-socket\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:26.231696 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.231478 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe123380-908c-44fd-bdd5-0c6b14972a1a-istio-podinfo\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:26.231696 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.231486 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s7zt\" (UniqueName: \"kubernetes.io/projected/fe123380-908c-44fd-bdd5-0c6b14972a1a-kube-api-access-5s7zt\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:26.288502 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.288470 2553 generic.go:356] "Generic (PLEG): container finished" podID="fe123380-908c-44fd-bdd5-0c6b14972a1a" containerID="e3b1bf0d672445713c5b45fda0aac395985a1625b7311abce94b30f9c268c135" exitCode=0 Sep 11 14:50:26.288662 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.288513 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" Sep 11 14:50:26.288662 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.288564 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" event={"ID":"fe123380-908c-44fd-bdd5-0c6b14972a1a","Type":"ContainerDied","Data":"e3b1bf0d672445713c5b45fda0aac395985a1625b7311abce94b30f9c268c135"} Sep 11 14:50:26.288662 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.288625 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6" event={"ID":"fe123380-908c-44fd-bdd5-0c6b14972a1a","Type":"ContainerDied","Data":"521652aa7f82c6acff638a5c1e55fa36ed576a73fea2469a8e127c53a640d1fc"} Sep 11 14:50:26.288662 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.288649 2553 scope.go:117] "RemoveContainer" containerID="e3b1bf0d672445713c5b45fda0aac395985a1625b7311abce94b30f9c268c135" Sep 11 14:50:26.298193 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.298177 2553 scope.go:117] "RemoveContainer" containerID="72502be9a5cbb58449d5ef51e580e0542291e6584b3b476c1845816a324be213" Sep 11 14:50:26.306218 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.306200 2553 scope.go:117] "RemoveContainer" containerID="e3b1bf0d672445713c5b45fda0aac395985a1625b7311abce94b30f9c268c135" Sep 11 14:50:26.306486 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:50:26.306466 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3b1bf0d672445713c5b45fda0aac395985a1625b7311abce94b30f9c268c135\": container with ID starting with e3b1bf0d672445713c5b45fda0aac395985a1625b7311abce94b30f9c268c135 not found: ID does not exist" containerID="e3b1bf0d672445713c5b45fda0aac395985a1625b7311abce94b30f9c268c135" Sep 11 14:50:26.306568 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.306507 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3b1bf0d672445713c5b45fda0aac395985a1625b7311abce94b30f9c268c135"} err="failed to get container status \"e3b1bf0d672445713c5b45fda0aac395985a1625b7311abce94b30f9c268c135\": rpc error: code = NotFound desc = could not find container \"e3b1bf0d672445713c5b45fda0aac395985a1625b7311abce94b30f9c268c135\": container with ID starting with e3b1bf0d672445713c5b45fda0aac395985a1625b7311abce94b30f9c268c135 not found: ID does not exist" Sep 11 14:50:26.306568 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.306530 2553 scope.go:117] "RemoveContainer" containerID="72502be9a5cbb58449d5ef51e580e0542291e6584b3b476c1845816a324be213" Sep 11 14:50:26.306769 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:50:26.306752 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72502be9a5cbb58449d5ef51e580e0542291e6584b3b476c1845816a324be213\": container with ID starting with 72502be9a5cbb58449d5ef51e580e0542291e6584b3b476c1845816a324be213 not found: ID does not exist" containerID="72502be9a5cbb58449d5ef51e580e0542291e6584b3b476c1845816a324be213" Sep 11 14:50:26.306810 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.306775 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72502be9a5cbb58449d5ef51e580e0542291e6584b3b476c1845816a324be213"} err="failed to get container status \"72502be9a5cbb58449d5ef51e580e0542291e6584b3b476c1845816a324be213\": rpc error: code = NotFound desc = could not find container \"72502be9a5cbb58449d5ef51e580e0542291e6584b3b476c1845816a324be213\": container with ID starting with 72502be9a5cbb58449d5ef51e580e0542291e6584b3b476c1845816a324be213 not found: ID does not exist" Sep 11 14:50:26.313030 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.313002 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6"] Sep 11 14:50:26.315653 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:26.315631 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-92ca78-openshift-default-d85d5577b-h42c6"] Sep 11 14:50:27.912461 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:27.912407 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe123380-908c-44fd-bdd5-0c6b14972a1a" path="/var/lib/kubelet/pods/fe123380-908c-44fd-bdd5-0c6b14972a1a/volumes" Sep 11 14:50:31.874812 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:31.874766 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:31.879964 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:31.879942 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8feb68-29818965-jtqsz"] Sep 11 14:50:31.882851 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:31.882826 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8feb68-29818965-jtqsz"] Sep 11 14:50:31.914879 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:31.914826 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd45bc23-fd9d-4dd0-bb7f-d824fee14fbe" path="/var/lib/kubelet/pods/cd45bc23-fd9d-4dd0-bb7f-d824fee14fbe/volumes" Sep 11 14:50:31.915369 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:31.915349 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z"] Sep 11 14:50:31.915756 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:31.915728 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z" podUID="50abaebd-162a-414d-9e3e-9c84ef2d30c1" containerName="payload-processing" containerID="cri-o://06d76460c26745d863d0164e0f335028fa7cbb81095dd0b6283fe0f8b9f1ed7a" gracePeriod=30 Sep 11 14:50:31.974653 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:31.974618 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:32.075210 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:32.075171 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:32.162178 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:32.162131 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:32.176880 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:32.176853 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z" Sep 11 14:50:32.288162 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:32.288113 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdhbk\" (UniqueName: \"kubernetes.io/projected/50abaebd-162a-414d-9e3e-9c84ef2d30c1-kube-api-access-sdhbk\") pod \"50abaebd-162a-414d-9e3e-9c84ef2d30c1\" (UID: \"50abaebd-162a-414d-9e3e-9c84ef2d30c1\") " Sep 11 14:50:32.288370 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:32.288193 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50abaebd-162a-414d-9e3e-9c84ef2d30c1-plugins-config-volume\") pod \"50abaebd-162a-414d-9e3e-9c84ef2d30c1\" (UID: \"50abaebd-162a-414d-9e3e-9c84ef2d30c1\") " Sep 11 14:50:32.288606 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:32.288580 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50abaebd-162a-414d-9e3e-9c84ef2d30c1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "50abaebd-162a-414d-9e3e-9c84ef2d30c1" (UID: "50abaebd-162a-414d-9e3e-9c84ef2d30c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:50:32.290280 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:32.290259 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50abaebd-162a-414d-9e3e-9c84ef2d30c1-kube-api-access-sdhbk" (OuterVolumeSpecName: "kube-api-access-sdhbk") pod "50abaebd-162a-414d-9e3e-9c84ef2d30c1" (UID: "50abaebd-162a-414d-9e3e-9c84ef2d30c1"). InnerVolumeSpecName "kube-api-access-sdhbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:50:32.320157 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:32.320121 2553 generic.go:356] "Generic (PLEG): container finished" podID="50abaebd-162a-414d-9e3e-9c84ef2d30c1" containerID="06d76460c26745d863d0164e0f335028fa7cbb81095dd0b6283fe0f8b9f1ed7a" exitCode=0 Sep 11 14:50:32.320301 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:32.320165 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z" Sep 11 14:50:32.320301 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:32.320196 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z" event={"ID":"50abaebd-162a-414d-9e3e-9c84ef2d30c1","Type":"ContainerDied","Data":"06d76460c26745d863d0164e0f335028fa7cbb81095dd0b6283fe0f8b9f1ed7a"} Sep 11 14:50:32.320301 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:32.320240 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z" event={"ID":"50abaebd-162a-414d-9e3e-9c84ef2d30c1","Type":"ContainerDied","Data":"f142bfa53f21641e438826c98d941f811c7298f6dc9b1ac5a8bd4a575ccf898d"} Sep 11 14:50:32.320301 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:32.320262 2553 scope.go:117] "RemoveContainer" containerID="06d76460c26745d863d0164e0f335028fa7cbb81095dd0b6283fe0f8b9f1ed7a" Sep 11 14:50:32.329078 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:32.329061 2553 scope.go:117] "RemoveContainer" containerID="06d76460c26745d863d0164e0f335028fa7cbb81095dd0b6283fe0f8b9f1ed7a" Sep 11 14:50:32.329335 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:50:32.329319 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06d76460c26745d863d0164e0f335028fa7cbb81095dd0b6283fe0f8b9f1ed7a\": container with ID starting with 06d76460c26745d863d0164e0f335028fa7cbb81095dd0b6283fe0f8b9f1ed7a not found: ID does not exist" containerID="06d76460c26745d863d0164e0f335028fa7cbb81095dd0b6283fe0f8b9f1ed7a" Sep 11 14:50:32.329389 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:32.329342 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d76460c26745d863d0164e0f335028fa7cbb81095dd0b6283fe0f8b9f1ed7a"} err="failed to get container status \"06d76460c26745d863d0164e0f335028fa7cbb81095dd0b6283fe0f8b9f1ed7a\": rpc error: code = NotFound desc = could not find container \"06d76460c26745d863d0164e0f335028fa7cbb81095dd0b6283fe0f8b9f1ed7a\": container with ID starting with 06d76460c26745d863d0164e0f335028fa7cbb81095dd0b6283fe0f8b9f1ed7a not found: ID does not exist" Sep 11 14:50:32.344730 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:32.344682 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z"] Sep 11 14:50:32.348197 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:32.348173 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8feb68-b844c89f7-dm87z"] Sep 11 14:50:32.389349 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:32.389319 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdhbk\" (UniqueName: \"kubernetes.io/projected/50abaebd-162a-414d-9e3e-9c84ef2d30c1-kube-api-access-sdhbk\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:32.389349 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:32.389346 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50abaebd-162a-414d-9e3e-9c84ef2d30c1-plugins-config-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:32.603970 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:32.603879 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:32.705773 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:32.705737 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:33.481596 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:33.481561 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:33.912704 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:33.912675 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50abaebd-162a-414d-9e3e-9c84ef2d30c1" path="/var/lib/kubelet/pods/50abaebd-162a-414d-9e3e-9c84ef2d30c1/volumes" Sep 11 14:50:39.622658 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:39.622622 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:39.729162 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:39.729127 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:39.898390 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:39.898350 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2"] Sep 11 14:50:39.898669 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:39.898637 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" podUID="b0ee52bc-2034-4487-8030-93c61a828509" containerName="istio-proxy" containerID="cri-o://097461543cfb31a8691986cfb07594d90c245552b59077a1e6a29249564bc546" gracePeriod=30 Sep 11 14:50:39.950603 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:39.950569 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:40.564568 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:40.564531 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:41.416638 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:41.416603 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:42.201763 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:42.201636 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:43.163223 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:43.163177 2553 patch_prober.go:28] interesting pod/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:50:43.163651 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:43.163239 2553 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" podUID="b0ee52bc-2034-4487-8030-93c61a828509" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:50:45.142748 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.142725 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:50:45.200701 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.200618 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-workload-socket\") pod \"b0ee52bc-2034-4487-8030-93c61a828509\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " Sep 11 14:50:45.200701 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.200655 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ee52bc-2034-4487-8030-93c61a828509-istiod-ca-cert\") pod \"b0ee52bc-2034-4487-8030-93c61a828509\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " Sep 11 14:50:45.200701 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.200690 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-istio-envoy\") pod \"b0ee52bc-2034-4487-8030-93c61a828509\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " Sep 11 14:50:45.200974 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.200725 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ee52bc-2034-4487-8030-93c61a828509-istio-podinfo\") pod \"b0ee52bc-2034-4487-8030-93c61a828509\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " Sep 11 14:50:45.200974 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.200786 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ee52bc-2034-4487-8030-93c61a828509-istio-token\") pod \"b0ee52bc-2034-4487-8030-93c61a828509\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " Sep 11 14:50:45.200974 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.200878 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b0ee52bc-2034-4487-8030-93c61a828509" (UID: "b0ee52bc-2034-4487-8030-93c61a828509"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:50:45.200974 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.200911 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-workload-certs\") pod \"b0ee52bc-2034-4487-8030-93c61a828509\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " Sep 11 14:50:45.200974 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.200954 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfmgt\" (UniqueName: \"kubernetes.io/projected/b0ee52bc-2034-4487-8030-93c61a828509-kube-api-access-kfmgt\") pod \"b0ee52bc-2034-4487-8030-93c61a828509\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " Sep 11 14:50:45.201232 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.200987 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-credential-socket\") pod \"b0ee52bc-2034-4487-8030-93c61a828509\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " Sep 11 14:50:45.201232 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.201013 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-istio-data\") pod \"b0ee52bc-2034-4487-8030-93c61a828509\" (UID: \"b0ee52bc-2034-4487-8030-93c61a828509\") " Sep 11 14:50:45.201232 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.201110 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ee52bc-2034-4487-8030-93c61a828509-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b0ee52bc-2034-4487-8030-93c61a828509" (UID: "b0ee52bc-2034-4487-8030-93c61a828509"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:50:45.201232 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.201209 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b0ee52bc-2034-4487-8030-93c61a828509" (UID: "b0ee52bc-2034-4487-8030-93c61a828509"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:50:45.201386 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.201233 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b0ee52bc-2034-4487-8030-93c61a828509" (UID: "b0ee52bc-2034-4487-8030-93c61a828509"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:50:45.201386 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.201363 2553 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-workload-certs\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:45.201386 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.201372 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-istio-data" (OuterVolumeSpecName: "istio-data") pod "b0ee52bc-2034-4487-8030-93c61a828509" (UID: "b0ee52bc-2034-4487-8030-93c61a828509"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:50:45.201386 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.201382 2553 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-credential-socket\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:45.201551 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.201396 2553 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-workload-socket\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:45.201551 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.201409 2553 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ee52bc-2034-4487-8030-93c61a828509-istiod-ca-cert\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:45.203045 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.203022 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b0ee52bc-2034-4487-8030-93c61a828509" (UID: "b0ee52bc-2034-4487-8030-93c61a828509"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:50:45.203245 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.203227 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ee52bc-2034-4487-8030-93c61a828509-kube-api-access-kfmgt" (OuterVolumeSpecName: "kube-api-access-kfmgt") pod "b0ee52bc-2034-4487-8030-93c61a828509" (UID: "b0ee52bc-2034-4487-8030-93c61a828509"). InnerVolumeSpecName "kube-api-access-kfmgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:50:45.203451 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.203393 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0ee52bc-2034-4487-8030-93c61a828509-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b0ee52bc-2034-4487-8030-93c61a828509" (UID: "b0ee52bc-2034-4487-8030-93c61a828509"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:50:45.203451 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.203402 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ee52bc-2034-4487-8030-93c61a828509-istio-token" (OuterVolumeSpecName: "istio-token") pod "b0ee52bc-2034-4487-8030-93c61a828509" (UID: "b0ee52bc-2034-4487-8030-93c61a828509"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:50:45.302083 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.302050 2553 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ee52bc-2034-4487-8030-93c61a828509-istio-token\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:45.302083 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.302076 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfmgt\" (UniqueName: \"kubernetes.io/projected/b0ee52bc-2034-4487-8030-93c61a828509-kube-api-access-kfmgt\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:45.302083 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.302085 2553 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-istio-data\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:45.302083 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.302094 2553 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ee52bc-2034-4487-8030-93c61a828509-istio-envoy\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:45.302350 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.302101 2553 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ee52bc-2034-4487-8030-93c61a828509-istio-podinfo\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:45.387083 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.387048 2553 generic.go:356] "Generic (PLEG): container finished" podID="b0ee52bc-2034-4487-8030-93c61a828509" containerID="097461543cfb31a8691986cfb07594d90c245552b59077a1e6a29249564bc546" exitCode=0 Sep 11 14:50:45.387228 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.387093 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" Sep 11 14:50:45.387228 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.387127 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" event={"ID":"b0ee52bc-2034-4487-8030-93c61a828509","Type":"ContainerDied","Data":"097461543cfb31a8691986cfb07594d90c245552b59077a1e6a29249564bc546"} Sep 11 14:50:45.387228 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.387181 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2" event={"ID":"b0ee52bc-2034-4487-8030-93c61a828509","Type":"ContainerDied","Data":"9d5464bcf73ab14a0dc5a91e8ac189f077dd909725883185899198d78e2646f2"} Sep 11 14:50:45.387228 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.387204 2553 scope.go:117] "RemoveContainer" containerID="097461543cfb31a8691986cfb07594d90c245552b59077a1e6a29249564bc546" Sep 11 14:50:45.396806 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.396787 2553 scope.go:117] "RemoveContainer" containerID="9178072f3c2888db7ef81da9171b03edefa570451c2ef893a088048fd2ce9ee9" Sep 11 14:50:45.404543 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.404523 2553 scope.go:117] "RemoveContainer" containerID="097461543cfb31a8691986cfb07594d90c245552b59077a1e6a29249564bc546" Sep 11 14:50:45.404821 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:50:45.404800 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"097461543cfb31a8691986cfb07594d90c245552b59077a1e6a29249564bc546\": container with ID starting with 097461543cfb31a8691986cfb07594d90c245552b59077a1e6a29249564bc546 not found: ID does not exist" containerID="097461543cfb31a8691986cfb07594d90c245552b59077a1e6a29249564bc546" Sep 11 14:50:45.404900 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.404834 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097461543cfb31a8691986cfb07594d90c245552b59077a1e6a29249564bc546"} err="failed to get container status \"097461543cfb31a8691986cfb07594d90c245552b59077a1e6a29249564bc546\": rpc error: code = NotFound desc = could not find container \"097461543cfb31a8691986cfb07594d90c245552b59077a1e6a29249564bc546\": container with ID starting with 097461543cfb31a8691986cfb07594d90c245552b59077a1e6a29249564bc546 not found: ID does not exist" Sep 11 14:50:45.404900 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.404870 2553 scope.go:117] "RemoveContainer" containerID="9178072f3c2888db7ef81da9171b03edefa570451c2ef893a088048fd2ce9ee9" Sep 11 14:50:45.405162 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:50:45.405143 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9178072f3c2888db7ef81da9171b03edefa570451c2ef893a088048fd2ce9ee9\": container with ID starting with 9178072f3c2888db7ef81da9171b03edefa570451c2ef893a088048fd2ce9ee9 not found: ID does not exist" containerID="9178072f3c2888db7ef81da9171b03edefa570451c2ef893a088048fd2ce9ee9" Sep 11 14:50:45.405208 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.405168 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9178072f3c2888db7ef81da9171b03edefa570451c2ef893a088048fd2ce9ee9"} err="failed to get container status \"9178072f3c2888db7ef81da9171b03edefa570451c2ef893a088048fd2ce9ee9\": rpc error: code = NotFound desc = could not find container \"9178072f3c2888db7ef81da9171b03edefa570451c2ef893a088048fd2ce9ee9\": container with ID starting with 9178072f3c2888db7ef81da9171b03edefa570451c2ef893a088048fd2ce9ee9 not found: ID does not exist" Sep 11 14:50:45.415425 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.415395 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2"] Sep 11 14:50:45.423591 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.423571 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8feb68-openshift-default-79fd879794-jphc2"] Sep 11 14:50:45.911372 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:45.911341 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0ee52bc-2034-4487-8030-93c61a828509" path="/var/lib/kubelet/pods/b0ee52bc-2034-4487-8030-93c61a828509/volumes" Sep 11 14:50:50.306405 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:50.306376 2553 scope.go:117] "RemoveContainer" containerID="65d8bd6c6cc2c1af6a7d8c8bcbd502cdd18d13d7c436d3c1b6bdd975834e49f1" Sep 11 14:50:50.976018 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:50.975978 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-977c12-29818965-kdz4b"] Sep 11 14:50:50.977840 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:50.977809 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-977c12-29818965-kdz4b"] Sep 11 14:50:51.002341 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.002306 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4"] Sep 11 14:50:51.002565 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.002537 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4" podUID="4fd81613-5178-4e10-8057-6cd282a6c4b7" containerName="payload-processing" containerID="cri-o://553390698f71674866d7a3075cd026749d0a3c30a91d3093b35b88ab0f7f2073" gracePeriod=30 Sep 11 14:50:51.031832 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.031806 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:51.218263 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.218233 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:51.264771 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.264744 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4" Sep 11 14:50:51.349511 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.349478 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4m7b6\" (UniqueName: \"kubernetes.io/projected/4fd81613-5178-4e10-8057-6cd282a6c4b7-kube-api-access-4m7b6\") pod \"4fd81613-5178-4e10-8057-6cd282a6c4b7\" (UID: \"4fd81613-5178-4e10-8057-6cd282a6c4b7\") " Sep 11 14:50:51.349980 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.349598 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fd81613-5178-4e10-8057-6cd282a6c4b7-plugins-config-volume\") pod \"4fd81613-5178-4e10-8057-6cd282a6c4b7\" (UID: \"4fd81613-5178-4e10-8057-6cd282a6c4b7\") " Sep 11 14:50:51.349980 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.349966 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd81613-5178-4e10-8057-6cd282a6c4b7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4fd81613-5178-4e10-8057-6cd282a6c4b7" (UID: "4fd81613-5178-4e10-8057-6cd282a6c4b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:50:51.351632 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.351608 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd81613-5178-4e10-8057-6cd282a6c4b7-kube-api-access-4m7b6" (OuterVolumeSpecName: "kube-api-access-4m7b6") pod "4fd81613-5178-4e10-8057-6cd282a6c4b7" (UID: "4fd81613-5178-4e10-8057-6cd282a6c4b7"). InnerVolumeSpecName "kube-api-access-4m7b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:50:51.401876 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.401845 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:51.419840 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.419805 2553 generic.go:356] "Generic (PLEG): container finished" podID="4fd81613-5178-4e10-8057-6cd282a6c4b7" containerID="553390698f71674866d7a3075cd026749d0a3c30a91d3093b35b88ab0f7f2073" exitCode=0 Sep 11 14:50:51.419984 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.419852 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4" Sep 11 14:50:51.419984 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.419891 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4" event={"ID":"4fd81613-5178-4e10-8057-6cd282a6c4b7","Type":"ContainerDied","Data":"553390698f71674866d7a3075cd026749d0a3c30a91d3093b35b88ab0f7f2073"} Sep 11 14:50:51.419984 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.419952 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4" event={"ID":"4fd81613-5178-4e10-8057-6cd282a6c4b7","Type":"ContainerDied","Data":"884c997b662e312d9e31e92c110e431594497c3826c31975e07833c5d4789b80"} Sep 11 14:50:51.419984 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.419978 2553 scope.go:117] "RemoveContainer" containerID="553390698f71674866d7a3075cd026749d0a3c30a91d3093b35b88ab0f7f2073" Sep 11 14:50:51.429229 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.429212 2553 scope.go:117] "RemoveContainer" containerID="553390698f71674866d7a3075cd026749d0a3c30a91d3093b35b88ab0f7f2073" Sep 11 14:50:51.429562 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:50:51.429539 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"553390698f71674866d7a3075cd026749d0a3c30a91d3093b35b88ab0f7f2073\": container with ID starting with 553390698f71674866d7a3075cd026749d0a3c30a91d3093b35b88ab0f7f2073 not found: ID does not exist" containerID="553390698f71674866d7a3075cd026749d0a3c30a91d3093b35b88ab0f7f2073" Sep 11 14:50:51.429640 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.429573 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"553390698f71674866d7a3075cd026749d0a3c30a91d3093b35b88ab0f7f2073"} err="failed to get container status \"553390698f71674866d7a3075cd026749d0a3c30a91d3093b35b88ab0f7f2073\": rpc error: code = NotFound desc = could not find container \"553390698f71674866d7a3075cd026749d0a3c30a91d3093b35b88ab0f7f2073\": container with ID starting with 553390698f71674866d7a3075cd026749d0a3c30a91d3093b35b88ab0f7f2073 not found: ID does not exist" Sep 11 14:50:51.443105 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.443079 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4"] Sep 11 14:50:51.450745 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.450712 2553 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fd81613-5178-4e10-8057-6cd282a6c4b7-plugins-config-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:51.450745 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.450736 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m7b6\" (UniqueName: \"kubernetes.io/projected/4fd81613-5178-4e10-8057-6cd282a6c4b7-kube-api-access-4m7b6\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:50:51.451265 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.451241 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-977c12-5bf7f46c-tgkg4"] Sep 11 14:50:51.501379 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.501297 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:51.750590 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.750557 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:51.850498 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.850380 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:51.917924 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.917889 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fd81613-5178-4e10-8057-6cd282a6c4b7" path="/var/lib/kubelet/pods/4fd81613-5178-4e10-8057-6cd282a6c4b7/volumes" Sep 11 14:50:51.918225 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:51.918207 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74a47a90-8e1d-4d0f-9aa9-2fb80b977711" path="/var/lib/kubelet/pods/74a47a90-8e1d-4d0f-9aa9-2fb80b977711/volumes" Sep 11 14:50:54.151690 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:54.151616 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:54.250262 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:54.250228 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:57.941748 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:57.941710 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:50:59.330675 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:50:59.330637 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:51:50.335838 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:51:50.335805 2553 scope.go:117] "RemoveContainer" containerID="2c00ab6043d36a902daca61cffa880683061a8da389419a6850bb2dff30f43bb" Sep 11 14:51:50.343896 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:51:50.343878 2553 scope.go:117] "RemoveContainer" containerID="c883c73f430bf9ed5677b9eb04427c06a516085b58dc5438c0dfb921904f839c" Sep 11 14:51:50.350981 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:51:50.350954 2553 scope.go:117] "RemoveContainer" containerID="142ef67051fa296482b4e6536e1995eeba1c8e62d63ee65b190de25484af33b9" Sep 11 14:51:52.747620 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:51:52.747586 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:51:52.831852 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:51:52.831822 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:52:22.603796 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:52:22.603757 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:52:24.332564 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:52:24.332533 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:52:50.379311 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:52:50.379277 2553 scope.go:117] "RemoveContainer" containerID="13ce19a2cdf891f648f84808f7affdf458390b519533f98ccece5e4a28c07cc6" Sep 11 14:53:29.895194 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:53:29.895154 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:53:36.015349 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:53:36.015313 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp"] Sep 11 14:53:36.015788 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:53:36.015549 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp" podUID="5e963db9-4b7b-41dc-b53b-72117b72a478" containerName="manager" containerID="cri-o://b6b3866c99ec96434f2b7724151a94296d0141095416a15b5c251f90c799d83a" gracePeriod=10 Sep 11 14:53:36.562258 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:53:36.562232 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp" Sep 11 14:53:36.657619 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:53:36.657589 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e963db9-4b7b-41dc-b53b-72117b72a478-extensions-socket-volume\") pod \"5e963db9-4b7b-41dc-b53b-72117b72a478\" (UID: \"5e963db9-4b7b-41dc-b53b-72117b72a478\") " Sep 11 14:53:36.657800 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:53:36.657627 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmgxt\" (UniqueName: \"kubernetes.io/projected/5e963db9-4b7b-41dc-b53b-72117b72a478-kube-api-access-tmgxt\") pod \"5e963db9-4b7b-41dc-b53b-72117b72a478\" (UID: \"5e963db9-4b7b-41dc-b53b-72117b72a478\") " Sep 11 14:53:36.657970 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:53:36.657943 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e963db9-4b7b-41dc-b53b-72117b72a478-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5e963db9-4b7b-41dc-b53b-72117b72a478" (UID: "5e963db9-4b7b-41dc-b53b-72117b72a478"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:53:36.659609 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:53:36.659585 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e963db9-4b7b-41dc-b53b-72117b72a478-kube-api-access-tmgxt" (OuterVolumeSpecName: "kube-api-access-tmgxt") pod "5e963db9-4b7b-41dc-b53b-72117b72a478" (UID: "5e963db9-4b7b-41dc-b53b-72117b72a478"). InnerVolumeSpecName "kube-api-access-tmgxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:53:36.759110 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:53:36.759065 2553 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e963db9-4b7b-41dc-b53b-72117b72a478-extensions-socket-volume\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:53:36.759110 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:53:36.759103 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmgxt\" (UniqueName: \"kubernetes.io/projected/5e963db9-4b7b-41dc-b53b-72117b72a478-kube-api-access-tmgxt\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:53:37.204283 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:53:37.204234 2553 generic.go:356] "Generic (PLEG): container finished" podID="5e963db9-4b7b-41dc-b53b-72117b72a478" containerID="b6b3866c99ec96434f2b7724151a94296d0141095416a15b5c251f90c799d83a" exitCode=0 Sep 11 14:53:37.204780 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:53:37.204349 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp" event={"ID":"5e963db9-4b7b-41dc-b53b-72117b72a478","Type":"ContainerDied","Data":"b6b3866c99ec96434f2b7724151a94296d0141095416a15b5c251f90c799d83a"} Sep 11 14:53:37.204780 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:53:37.204401 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp" event={"ID":"5e963db9-4b7b-41dc-b53b-72117b72a478","Type":"ContainerDied","Data":"9538d74f56bf16216c4987ad7387daad4ce2cdd2efb3788986bff5ff2fdaea08"} Sep 11 14:53:37.204780 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:53:37.204456 2553 scope.go:117] "RemoveContainer" containerID="b6b3866c99ec96434f2b7724151a94296d0141095416a15b5c251f90c799d83a" Sep 11 14:53:37.204780 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:53:37.204568 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp" Sep 11 14:53:37.216335 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:53:37.216316 2553 scope.go:117] "RemoveContainer" containerID="b6b3866c99ec96434f2b7724151a94296d0141095416a15b5c251f90c799d83a" Sep 11 14:53:37.216679 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:53:37.216659 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6b3866c99ec96434f2b7724151a94296d0141095416a15b5c251f90c799d83a\": container with ID starting with b6b3866c99ec96434f2b7724151a94296d0141095416a15b5c251f90c799d83a not found: ID does not exist" containerID="b6b3866c99ec96434f2b7724151a94296d0141095416a15b5c251f90c799d83a" Sep 11 14:53:37.216756 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:53:37.216693 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6b3866c99ec96434f2b7724151a94296d0141095416a15b5c251f90c799d83a"} err="failed to get container status \"b6b3866c99ec96434f2b7724151a94296d0141095416a15b5c251f90c799d83a\": rpc error: code = NotFound desc = could not find container \"b6b3866c99ec96434f2b7724151a94296d0141095416a15b5c251f90c799d83a\": container with ID starting with b6b3866c99ec96434f2b7724151a94296d0141095416a15b5c251f90c799d83a not found: ID does not exist" Sep 11 14:53:37.234401 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:53:37.234369 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp"] Sep 11 14:53:37.238568 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:53:37.238541 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5b9pp"] Sep 11 14:53:37.911366 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:53:37.911332 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e963db9-4b7b-41dc-b53b-72117b72a478" path="/var/lib/kubelet/pods/5e963db9-4b7b-41dc-b53b-72117b72a478/volumes" Sep 11 14:54:32.691680 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:32.691641 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:54:36.687137 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687106 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fc6sk"] Sep 11 14:54:36.687580 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687464 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec5616fa-696a-43b8-a294-cbcbdba5eb46" containerName="payload-pre-processing" Sep 11 14:54:36.687580 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687477 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec5616fa-696a-43b8-a294-cbcbdba5eb46" containerName="payload-pre-processing" Sep 11 14:54:36.687580 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687486 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed0f67bc-254a-4c16-bd37-d7b96c926784" containerName="payload-processing" Sep 11 14:54:36.687580 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687492 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed0f67bc-254a-4c16-bd37-d7b96c926784" containerName="payload-processing" Sep 11 14:54:36.687580 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687500 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe123380-908c-44fd-bdd5-0c6b14972a1a" containerName="istio-proxy" Sep 11 14:54:36.687580 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687505 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe123380-908c-44fd-bdd5-0c6b14972a1a" containerName="istio-proxy" Sep 11 14:54:36.687580 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687513 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c00c8577-0541-4e4f-8198-38cf7f5f2062" containerName="revoke-keys" Sep 11 14:54:36.687580 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687518 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c00c8577-0541-4e4f-8198-38cf7f5f2062" containerName="revoke-keys" Sep 11 14:54:36.687580 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687526 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0ee52bc-2034-4487-8030-93c61a828509" containerName="istio-proxy" Sep 11 14:54:36.687580 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687532 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0ee52bc-2034-4487-8030-93c61a828509" containerName="istio-proxy" Sep 11 14:54:36.687580 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687540 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0ee52bc-2034-4487-8030-93c61a828509" containerName="istio-proxy" Sep 11 14:54:36.687580 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687545 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0ee52bc-2034-4487-8030-93c61a828509" containerName="istio-proxy" Sep 11 14:54:36.687580 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687553 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe123380-908c-44fd-bdd5-0c6b14972a1a" containerName="istio-proxy" Sep 11 14:54:36.687580 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687558 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe123380-908c-44fd-bdd5-0c6b14972a1a" containerName="istio-proxy" Sep 11 14:54:36.687580 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687567 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e963db9-4b7b-41dc-b53b-72117b72a478" containerName="manager" Sep 11 14:54:36.687580 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687572 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e963db9-4b7b-41dc-b53b-72117b72a478" containerName="manager" Sep 11 14:54:36.687580 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687580 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fd81613-5178-4e10-8057-6cd282a6c4b7" containerName="payload-processing" Sep 11 14:54:36.687580 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687586 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fd81613-5178-4e10-8057-6cd282a6c4b7" containerName="payload-processing" Sep 11 14:54:36.687580 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687593 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50abaebd-162a-414d-9e3e-9c84ef2d30c1" containerName="payload-processing" Sep 11 14:54:36.688517 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687602 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="50abaebd-162a-414d-9e3e-9c84ef2d30c1" containerName="payload-processing" Sep 11 14:54:36.688517 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687656 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fd81613-5178-4e10-8057-6cd282a6c4b7" containerName="payload-processing" Sep 11 14:54:36.688517 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687664 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec5616fa-696a-43b8-a294-cbcbdba5eb46" containerName="payload-pre-processing" Sep 11 14:54:36.688517 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687670 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0ee52bc-2034-4487-8030-93c61a828509" containerName="istio-proxy" Sep 11 14:54:36.688517 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687676 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c00c8577-0541-4e4f-8198-38cf7f5f2062" containerName="revoke-keys" Sep 11 14:54:36.688517 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687681 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed0f67bc-254a-4c16-bd37-d7b96c926784" containerName="payload-processing" Sep 11 14:54:36.688517 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687686 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe123380-908c-44fd-bdd5-0c6b14972a1a" containerName="istio-proxy" Sep 11 14:54:36.688517 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687692 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe123380-908c-44fd-bdd5-0c6b14972a1a" containerName="istio-proxy" Sep 11 14:54:36.688517 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687699 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e963db9-4b7b-41dc-b53b-72117b72a478" containerName="manager" Sep 11 14:54:36.688517 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.687704 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="50abaebd-162a-414d-9e3e-9c84ef2d30c1" containerName="payload-processing" Sep 11 14:54:36.690762 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.690745 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fc6sk" Sep 11 14:54:36.693539 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.693523 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6dd7k\"" Sep 11 14:54:36.701180 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.701160 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fc6sk"] Sep 11 14:54:36.800563 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.800523 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/21650479-3518-4f89-a31f-5f4028ebf63d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fc6sk\" (UID: \"21650479-3518-4f89-a31f-5f4028ebf63d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fc6sk" Sep 11 14:54:36.800563 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.800567 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79fhh\" (UniqueName: \"kubernetes.io/projected/21650479-3518-4f89-a31f-5f4028ebf63d-kube-api-access-79fhh\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fc6sk\" (UID: \"21650479-3518-4f89-a31f-5f4028ebf63d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fc6sk" Sep 11 14:54:36.901886 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.901837 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/21650479-3518-4f89-a31f-5f4028ebf63d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fc6sk\" (UID: \"21650479-3518-4f89-a31f-5f4028ebf63d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fc6sk" Sep 11 14:54:36.901886 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.901892 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79fhh\" (UniqueName: \"kubernetes.io/projected/21650479-3518-4f89-a31f-5f4028ebf63d-kube-api-access-79fhh\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fc6sk\" (UID: \"21650479-3518-4f89-a31f-5f4028ebf63d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fc6sk" Sep 11 14:54:36.902223 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.902204 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/21650479-3518-4f89-a31f-5f4028ebf63d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fc6sk\" (UID: \"21650479-3518-4f89-a31f-5f4028ebf63d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fc6sk" Sep 11 14:54:36.910535 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:36.910512 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79fhh\" (UniqueName: \"kubernetes.io/projected/21650479-3518-4f89-a31f-5f4028ebf63d-kube-api-access-79fhh\") pod \"kuadrant-operator-controller-manager-5769f99d9f-fc6sk\" (UID: \"21650479-3518-4f89-a31f-5f4028ebf63d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fc6sk" Sep 11 14:54:37.001404 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:37.001304 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fc6sk" Sep 11 14:54:37.276516 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:37.276493 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fc6sk"] Sep 11 14:54:37.278631 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:54:37.278604 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21650479_3518_4f89_a31f_5f4028ebf63d.slice/crio-365acc99298d61c60c39c04e3cab890c03c5466d31528def32ae9042fde04a98 WatchSource:0}: Error finding container 365acc99298d61c60c39c04e3cab890c03c5466d31528def32ae9042fde04a98: Status 404 returned error can't find the container with id 365acc99298d61c60c39c04e3cab890c03c5466d31528def32ae9042fde04a98 Sep 11 14:54:37.487914 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:37.487879 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fc6sk" event={"ID":"21650479-3518-4f89-a31f-5f4028ebf63d","Type":"ContainerStarted","Data":"fd928b28572c4c338a8460b28415add3b5e3dafcff159a2744c33269efdf03d1"} Sep 11 14:54:37.488129 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:37.487930 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fc6sk" event={"ID":"21650479-3518-4f89-a31f-5f4028ebf63d","Type":"ContainerStarted","Data":"365acc99298d61c60c39c04e3cab890c03c5466d31528def32ae9042fde04a98"} Sep 11 14:54:37.488129 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:37.487961 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fc6sk" Sep 11 14:54:37.507926 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:37.507694 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fc6sk" podStartSLOduration=1.507675179 podStartE2EDuration="1.507675179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:54:37.506371662 +0000 UTC m=+2388.120478006" watchObservedRunningTime="2026-09-11 14:54:37.507675179 +0000 UTC m=+2388.121781528" Sep 11 14:54:42.243826 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:42.243792 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:54:48.494849 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:48.494814 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-fc6sk" Sep 11 14:54:50.066801 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:50.066771 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmqjh_6b2452d9-1730-458c-b380-dbcfd9e5be19/ovn-acl-logging/0.log" Sep 11 14:54:50.071020 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:54:50.070999 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-mmqjh_6b2452d9-1730-458c-b380-dbcfd9e5be19/ovn-acl-logging/0.log" Sep 11 14:55:36.214311 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:36.214275 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:55:46.380792 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.380752 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff"] Sep 11 14:55:46.381465 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.381444 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0ee52bc-2034-4487-8030-93c61a828509" containerName="istio-proxy" Sep 11 14:55:46.385031 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.385012 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:46.388395 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.388376 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-7350e86d-kserve-self-signed-certs\"" Sep 11 14:55:46.397708 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.397687 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff"] Sep 11 14:55:46.471819 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.471786 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:55:46.543904 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.543868 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa035785-398d-4846-b444-555805cebbaf-tls-certs\") pod \"e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:46.544097 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.543972 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-model-cache\") pod \"e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:46.544097 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.543997 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-628dd\" (UniqueName: \"kubernetes.io/projected/fa035785-398d-4846-b444-555805cebbaf-kube-api-access-628dd\") pod \"e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:46.544097 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.544018 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-home\") pod \"e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:46.544225 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.544094 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-dshm\") pod \"e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:46.544225 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.544135 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-tmp-dir\") pod \"e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:46.645054 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.645026 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa035785-398d-4846-b444-555805cebbaf-tls-certs\") pod \"e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:46.645238 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.645090 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-model-cache\") pod \"e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:46.645238 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.645115 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-628dd\" (UniqueName: \"kubernetes.io/projected/fa035785-398d-4846-b444-555805cebbaf-kube-api-access-628dd\") pod \"e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:46.645238 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.645134 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-home\") pod \"e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:46.645238 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.645150 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-dshm\") pod \"e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:46.645238 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.645172 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-tmp-dir\") pod \"e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:46.645658 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.645626 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-home\") pod \"e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:46.645743 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.645660 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-model-cache\") pod \"e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:46.645743 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.645703 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-tmp-dir\") pod \"e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:46.647502 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.647481 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-dshm\") pod \"e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:46.647751 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.647732 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa035785-398d-4846-b444-555805cebbaf-tls-certs\") pod \"e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:46.661803 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.661773 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-628dd\" (UniqueName: \"kubernetes.io/projected/fa035785-398d-4846-b444-555805cebbaf-kube-api-access-628dd\") pod \"e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:46.694665 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.694640 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:46.971222 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:46.971194 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff"] Sep 11 14:55:46.973299 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:55:46.973271 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa035785_398d_4846_b444_555805cebbaf.slice/crio-931ab11f78f205da2be9b37f98f9eb3439ef14253932da9dedaca1930c6cc1ec WatchSource:0}: Error finding container 931ab11f78f205da2be9b37f98f9eb3439ef14253932da9dedaca1930c6cc1ec: Status 404 returned error can't find the container with id 931ab11f78f205da2be9b37f98f9eb3439ef14253932da9dedaca1930c6cc1ec Sep 11 14:55:47.816891 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:47.816855 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" event={"ID":"fa035785-398d-4846-b444-555805cebbaf","Type":"ContainerStarted","Data":"38c0600bcc658b645751f6a8022bd7850b713bc5937b22d36ad8542018f0b8e4"} Sep 11 14:55:47.816891 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:47.816898 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" event={"ID":"fa035785-398d-4846-b444-555805cebbaf","Type":"ContainerStarted","Data":"931ab11f78f205da2be9b37f98f9eb3439ef14253932da9dedaca1930c6cc1ec"} Sep 11 14:55:47.835521 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:47.835456 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" podStartSLOduration=1.835435962 podStartE2EDuration="1.835435962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:55:47.833247967 +0000 UTC m=+2458.447354313" watchObservedRunningTime="2026-09-11 14:55:47.835435962 +0000 UTC m=+2458.449542303" Sep 11 14:55:50.416677 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:50.416635 2553 scope.go:117] "RemoveContainer" containerID="dc34a169e4a2778e71252d8abb66b518f7c9e2fe87c30c9e856679c6b0a57e20" Sep 11 14:55:56.695370 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:56.695324 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:56.695370 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:56.695374 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:56.707956 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:56.707934 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:55:56.877209 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:55:56.877179 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:56:13.604100 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:13.604067 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:56:14.495559 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:14.495529 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff"] Sep 11 14:56:15.020512 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:15.020476 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:56:29.514013 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:29.513949 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" podUID="fa035785-398d-4846-b444-555805cebbaf" containerName="main" containerID="cri-o://38c0600bcc658b645751f6a8022bd7850b713bc5937b22d36ad8542018f0b8e4" gracePeriod=45 Sep 11 14:56:29.754789 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:29.754757 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:56:29.848273 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:29.848181 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa035785-398d-4846-b444-555805cebbaf-tls-certs\") pod \"fa035785-398d-4846-b444-555805cebbaf\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " Sep 11 14:56:29.848273 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:29.848240 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-home\") pod \"fa035785-398d-4846-b444-555805cebbaf\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " Sep 11 14:56:29.848273 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:29.848271 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-628dd\" (UniqueName: \"kubernetes.io/projected/fa035785-398d-4846-b444-555805cebbaf-kube-api-access-628dd\") pod \"fa035785-398d-4846-b444-555805cebbaf\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " Sep 11 14:56:29.848610 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:29.848305 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-tmp-dir\") pod \"fa035785-398d-4846-b444-555805cebbaf\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " Sep 11 14:56:29.848610 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:29.848337 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-model-cache\") pod \"fa035785-398d-4846-b444-555805cebbaf\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " Sep 11 14:56:29.848610 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:29.848393 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-dshm\") pod \"fa035785-398d-4846-b444-555805cebbaf\" (UID: \"fa035785-398d-4846-b444-555805cebbaf\") " Sep 11 14:56:29.848610 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:29.848550 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-home" (OuterVolumeSpecName: "home") pod "fa035785-398d-4846-b444-555805cebbaf" (UID: "fa035785-398d-4846-b444-555805cebbaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:56:29.848835 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:29.848642 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa035785-398d-4846-b444-555805cebbaf" (UID: "fa035785-398d-4846-b444-555805cebbaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:56:29.848835 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:29.848664 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa035785-398d-4846-b444-555805cebbaf" (UID: "fa035785-398d-4846-b444-555805cebbaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:56:29.849281 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:29.848810 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-home\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:56:29.849394 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:29.849289 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-tmp-dir\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:56:29.849394 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:29.849306 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-model-cache\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:56:29.857507 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:29.857473 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa035785-398d-4846-b444-555805cebbaf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa035785-398d-4846-b444-555805cebbaf" (UID: "fa035785-398d-4846-b444-555805cebbaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:56:29.857633 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:29.857494 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa035785-398d-4846-b444-555805cebbaf-kube-api-access-628dd" (OuterVolumeSpecName: "kube-api-access-628dd") pod "fa035785-398d-4846-b444-555805cebbaf" (UID: "fa035785-398d-4846-b444-555805cebbaf"). InnerVolumeSpecName "kube-api-access-628dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:56:29.857824 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:29.857789 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-dshm" (OuterVolumeSpecName: "dshm") pod "fa035785-398d-4846-b444-555805cebbaf" (UID: "fa035785-398d-4846-b444-555805cebbaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:56:29.950447 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:29.950395 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa035785-398d-4846-b444-555805cebbaf-tls-certs\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:56:29.950447 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:29.950447 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-628dd\" (UniqueName: \"kubernetes.io/projected/fa035785-398d-4846-b444-555805cebbaf-kube-api-access-628dd\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:56:29.950636 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:29.950462 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa035785-398d-4846-b444-555805cebbaf-dshm\") on node \"ip-10-0-135-17.ec2.internal\" DevicePath \"\"" Sep 11 14:56:30.023182 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:30.023142 2553 generic.go:356] "Generic (PLEG): container finished" podID="fa035785-398d-4846-b444-555805cebbaf" containerID="38c0600bcc658b645751f6a8022bd7850b713bc5937b22d36ad8542018f0b8e4" exitCode=0 Sep 11 14:56:30.023344 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:30.023202 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" Sep 11 14:56:30.023344 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:30.023232 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" event={"ID":"fa035785-398d-4846-b444-555805cebbaf","Type":"ContainerDied","Data":"38c0600bcc658b645751f6a8022bd7850b713bc5937b22d36ad8542018f0b8e4"} Sep 11 14:56:30.023344 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:30.023278 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff" event={"ID":"fa035785-398d-4846-b444-555805cebbaf","Type":"ContainerDied","Data":"931ab11f78f205da2be9b37f98f9eb3439ef14253932da9dedaca1930c6cc1ec"} Sep 11 14:56:30.023344 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:30.023299 2553 scope.go:117] "RemoveContainer" containerID="38c0600bcc658b645751f6a8022bd7850b713bc5937b22d36ad8542018f0b8e4" Sep 11 14:56:30.031665 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:30.031649 2553 scope.go:117] "RemoveContainer" containerID="38c0600bcc658b645751f6a8022bd7850b713bc5937b22d36ad8542018f0b8e4" Sep 11 14:56:30.031955 ip-10-0-135-17 kubenswrapper[2553]: E0911 14:56:30.031929 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38c0600bcc658b645751f6a8022bd7850b713bc5937b22d36ad8542018f0b8e4\": container with ID starting with 38c0600bcc658b645751f6a8022bd7850b713bc5937b22d36ad8542018f0b8e4 not found: ID does not exist" containerID="38c0600bcc658b645751f6a8022bd7850b713bc5937b22d36ad8542018f0b8e4" Sep 11 14:56:30.032049 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:30.031965 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38c0600bcc658b645751f6a8022bd7850b713bc5937b22d36ad8542018f0b8e4"} err="failed to get container status \"38c0600bcc658b645751f6a8022bd7850b713bc5937b22d36ad8542018f0b8e4\": rpc error: code = NotFound desc = could not find container \"38c0600bcc658b645751f6a8022bd7850b713bc5937b22d36ad8542018f0b8e4\": container with ID starting with 38c0600bcc658b645751f6a8022bd7850b713bc5937b22d36ad8542018f0b8e4 not found: ID does not exist" Sep 11 14:56:30.040257 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:30.040234 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff"] Sep 11 14:56:30.043771 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:30.043752 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-7350e86d-kserve-79fb55cf75-6k6ff"] Sep 11 14:56:31.911693 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:56:31.911657 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa035785-398d-4846-b444-555805cebbaf" path="/var/lib/kubelet/pods/fa035785-398d-4846-b444-555805cebbaf/volumes" Sep 11 14:57:01.318093 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:01.318051 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5645d84896-j49m8_f5ee2a26-df9b-4907-80e6-12c357c2fbb3/authorino/0.log" Sep 11 14:57:07.444650 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:07.444624 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86d7f488b8-8xdsr_e018c54d-e91d-4b5d-997f-cf8884c3cac9/copy-manifests/0.log" Sep 11 14:57:07.454103 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:07.454067 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86d7f488b8-8xdsr_e018c54d-e91d-4b5d-997f-cf8884c3cac9/manager/0.log" Sep 11 14:57:09.233571 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:09.233538 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5645d84896-j49m8_f5ee2a26-df9b-4907-80e6-12c357c2fbb3/authorino/0.log" Sep 11 14:57:09.723043 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:09.723013 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-fc6sk_21650479-3518-4f89-a31f-5f4028ebf63d/manager/0.log" Sep 11 14:57:10.578618 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:10.578587 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-769cf48dc6-74wxb_f40b578b-ce2a-4eca-96c9-0f88c9e90261/kube-auth-proxy/0.log" Sep 11 14:57:11.054679 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:11.054647 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-cvn67_35517420-4a51-4512-810b-834bb79de2f0/payload-processing/0.log" Sep 11 14:57:11.609993 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:11.609944 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-9865t_40dbd1f7-2e0b-4fcd-a2e0-83b618a6b1e6/main/0.log" Sep 11 14:57:11.868642 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:11.868542 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-mzggg_a07a6e3b-7be3-4bb3-abb8-465dfd2b0ae5/main/0.log" Sep 11 14:57:12.226045 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:12.226011 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-nhp9t_7a17fe65-0779-4a4d-99aa-d9bb1fc58ba6/main/0.log" Sep 11 14:57:17.827889 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:17.827854 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4bt5c/must-gather-m24hw"] Sep 11 14:57:17.828279 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:17.828220 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa035785-398d-4846-b444-555805cebbaf" containerName="main" Sep 11 14:57:17.828279 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:17.828232 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa035785-398d-4846-b444-555805cebbaf" containerName="main" Sep 11 14:57:17.828353 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:17.828296 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa035785-398d-4846-b444-555805cebbaf" containerName="main" Sep 11 14:57:17.831539 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:17.831519 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4bt5c/must-gather-m24hw" Sep 11 14:57:17.832689 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:17.832648 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:57:17.834337 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:17.834314 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4bt5c\"/\"kube-root-ca.crt\"" Sep 11 14:57:17.834460 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:17.834371 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4bt5c\"/\"openshift-service-ca.crt\"" Sep 11 14:57:17.834460 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:17.834382 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4bt5c\"/\"default-dockercfg-2wjmk\"" Sep 11 14:57:17.852197 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:17.852170 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4bt5c/must-gather-m24hw"] Sep 11 14:57:17.931431 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:17.931385 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:57:17.979782 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:17.979753 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/05c32fa1-7529-44e9-ae7a-d226d9229efb-must-gather-output\") pod \"must-gather-m24hw\" (UID: \"05c32fa1-7529-44e9-ae7a-d226d9229efb\") " pod="openshift-must-gather-4bt5c/must-gather-m24hw" Sep 11 14:57:17.979948 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:17.979811 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gfn8\" (UniqueName: \"kubernetes.io/projected/05c32fa1-7529-44e9-ae7a-d226d9229efb-kube-api-access-6gfn8\") pod \"must-gather-m24hw\" (UID: \"05c32fa1-7529-44e9-ae7a-d226d9229efb\") " pod="openshift-must-gather-4bt5c/must-gather-m24hw" Sep 11 14:57:18.081122 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:18.081027 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/05c32fa1-7529-44e9-ae7a-d226d9229efb-must-gather-output\") pod \"must-gather-m24hw\" (UID: \"05c32fa1-7529-44e9-ae7a-d226d9229efb\") " pod="openshift-must-gather-4bt5c/must-gather-m24hw" Sep 11 14:57:18.081279 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:18.081183 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gfn8\" (UniqueName: \"kubernetes.io/projected/05c32fa1-7529-44e9-ae7a-d226d9229efb-kube-api-access-6gfn8\") pod \"must-gather-m24hw\" (UID: \"05c32fa1-7529-44e9-ae7a-d226d9229efb\") " pod="openshift-must-gather-4bt5c/must-gather-m24hw" Sep 11 14:57:18.081370 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:18.081350 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/05c32fa1-7529-44e9-ae7a-d226d9229efb-must-gather-output\") pod \"must-gather-m24hw\" (UID: \"05c32fa1-7529-44e9-ae7a-d226d9229efb\") " pod="openshift-must-gather-4bt5c/must-gather-m24hw" Sep 11 14:57:18.090925 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:18.090905 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gfn8\" (UniqueName: \"kubernetes.io/projected/05c32fa1-7529-44e9-ae7a-d226d9229efb-kube-api-access-6gfn8\") pod \"must-gather-m24hw\" (UID: \"05c32fa1-7529-44e9-ae7a-d226d9229efb\") " pod="openshift-must-gather-4bt5c/must-gather-m24hw" Sep 11 14:57:18.141691 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:18.141652 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4bt5c/must-gather-m24hw" Sep 11 14:57:18.416558 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:18.416532 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4bt5c/must-gather-m24hw"] Sep 11 14:57:18.418493 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:57:18.418465 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05c32fa1_7529_44e9_ae7a_d226d9229efb.slice/crio-9bc7676cbf79a7bb71ed86e867f71a201b02d50af1a1462f4463d356a8450679 WatchSource:0}: Error finding container 9bc7676cbf79a7bb71ed86e867f71a201b02d50af1a1462f4463d356a8450679: Status 404 returned error can't find the container with id 9bc7676cbf79a7bb71ed86e867f71a201b02d50af1a1462f4463d356a8450679 Sep 11 14:57:18.420747 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:18.420730 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:57:19.250165 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:19.250139 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4bt5c/must-gather-m24hw" event={"ID":"05c32fa1-7529-44e9-ae7a-d226d9229efb","Type":"ContainerStarted","Data":"9bc7676cbf79a7bb71ed86e867f71a201b02d50af1a1462f4463d356a8450679"} Sep 11 14:57:20.259564 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:20.259518 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4bt5c/must-gather-m24hw" event={"ID":"05c32fa1-7529-44e9-ae7a-d226d9229efb","Type":"ContainerStarted","Data":"91e3c62a9f7de2d9227ba86830a42a937672bed01edae9f31a162085b74090f3"} Sep 11 14:57:20.260043 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:20.259577 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4bt5c/must-gather-m24hw" event={"ID":"05c32fa1-7529-44e9-ae7a-d226d9229efb","Type":"ContainerStarted","Data":"466674a297cccc9c091c79295dcae887ce8f42e71e01305cc5b1592f672d804c"} Sep 11 14:57:20.278354 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:20.278290 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4bt5c/must-gather-m24hw" podStartSLOduration=2.45341391 podStartE2EDuration="3.278271014s" totalImagesPullingTime="824.857104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:57:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:57:20.27774267 +0000 UTC m=+2550.891849016" watchObservedRunningTime="2026-09-11 14:57:20.278271014 +0000 UTC m=+2550.892377357" Sep 11 14:57:20.975206 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:20.975163 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-n7x6t_bbb4cb01-8e27-4823-9410-d5b11334c3a9/global-pull-secret-syncer/0.log" Sep 11 14:57:21.092938 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:21.092905 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" Sep 11 14:57:21.218004 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:21.217965 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-17.ec2.internal_7c73b8b4f6cacbe91fc9321bf2769db6/haproxy/0.log" Sep 11 14:57:25.732585 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:25.732545 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5645d84896-j49m8_f5ee2a26-df9b-4907-80e6-12c357c2fbb3/authorino/0.log" Sep 11 14:57:25.901237 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:25.901196 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-fc6sk_21650479-3518-4f89-a31f-5f4028ebf63d/manager/0.log" Sep 11 14:57:27.599479 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:27.599448 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_35b3de7c-77d1-4b82-9a03-c846fb066bda/alertmanager/0.log" Sep 11 14:57:27.634283 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:27.634248 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_35b3de7c-77d1-4b82-9a03-c846fb066bda/config-reloader/0.log" Sep 11 14:57:27.662989 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:27.662902 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_35b3de7c-77d1-4b82-9a03-c846fb066bda/kube-rbac-proxy-web/0.log" Sep 11 14:57:27.697425 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:27.697384 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_35b3de7c-77d1-4b82-9a03-c846fb066bda/kube-rbac-proxy/0.log" Sep 11 14:57:27.727104 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:27.727071 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_35b3de7c-77d1-4b82-9a03-c846fb066bda/kube-rbac-proxy-metric/0.log" Sep 11 14:57:27.763310 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:27.763282 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_35b3de7c-77d1-4b82-9a03-c846fb066bda/prom-label-proxy/0.log" Sep 11 14:57:27.787878 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:27.787784 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_35b3de7c-77d1-4b82-9a03-c846fb066bda/init-config-reloader/0.log" Sep 11 14:57:28.189445 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:28.189401 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kgzrk_11604680-a734-4e85-81e3-8dc215a6fd4a/node-exporter/0.log" Sep 11 14:57:28.218830 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:28.218802 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kgzrk_11604680-a734-4e85-81e3-8dc215a6fd4a/kube-rbac-proxy/0.log" Sep 11 14:57:28.247552 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:28.247518 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kgzrk_11604680-a734-4e85-81e3-8dc215a6fd4a/init-textfile/0.log" Sep 11 14:57:28.381734 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:28.381700 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-cx7cp_95c23d32-5f8b-4fbd-8e68-dd65447d578d/kube-rbac-proxy-main/0.log" Sep 11 14:57:28.423791 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:28.423760 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-cx7cp_95c23d32-5f8b-4fbd-8e68-dd65447d578d/kube-rbac-proxy-self/0.log" Sep 11 14:57:28.466966 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:28.466878 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-cx7cp_95c23d32-5f8b-4fbd-8e68-dd65447d578d/openshift-state-metrics/0.log" Sep 11 14:57:28.535607 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:28.535574 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7637927e-8707-48f3-afaf-d64d2c4c4b38/prometheus/0.log" Sep 11 14:57:28.619110 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:28.619078 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7637927e-8707-48f3-afaf-d64d2c4c4b38/config-reloader/0.log" Sep 11 14:57:28.672146 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:28.672118 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7637927e-8707-48f3-afaf-d64d2c4c4b38/thanos-sidecar/0.log" Sep 11 14:57:28.722437 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:28.722332 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7637927e-8707-48f3-afaf-d64d2c4c4b38/kube-rbac-proxy-web/0.log" Sep 11 14:57:28.766325 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:28.766287 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7637927e-8707-48f3-afaf-d64d2c4c4b38/kube-rbac-proxy/0.log" Sep 11 14:57:28.801838 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:28.801806 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7637927e-8707-48f3-afaf-d64d2c4c4b38/kube-rbac-proxy-thanos/0.log" Sep 11 14:57:28.833035 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:28.832976 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7637927e-8707-48f3-afaf-d64d2c4c4b38/init-config-reloader/0.log" Sep 11 14:57:28.934268 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:28.934239 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-pd78h_c79d8dda-9f21-42df-a4fa-e350d0d381bb/prometheus-operator-admission-webhook/0.log" Sep 11 14:57:28.975385 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:28.975305 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6486f4694f-x8bc4_16b90586-3aae-43c1-8b1b-40a81db48e7e/telemeter-client/0.log" Sep 11 14:57:28.998639 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:28.998611 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6486f4694f-x8bc4_16b90586-3aae-43c1-8b1b-40a81db48e7e/reload/0.log" Sep 11 14:57:29.025501 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:29.025382 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6486f4694f-x8bc4_16b90586-3aae-43c1-8b1b-40a81db48e7e/kube-rbac-proxy/0.log" Sep 11 14:57:30.499679 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:30.499641 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-59phf_c35b665e-20f7-4c00-a220-5b605f5f540d/networking-console-plugin/0.log" Sep 11 14:57:31.024504 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:31.024468 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r"] Sep 11 14:57:31.030659 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:31.030632 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r" Sep 11 14:57:31.037203 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:31.037173 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r"] Sep 11 14:57:31.120934 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:31.120893 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6f7q\" (UniqueName: \"kubernetes.io/projected/711b7600-aba8-428e-aa8a-c661e05216d6-kube-api-access-p6f7q\") pod \"perf-node-gather-daemonset-52l6r\" (UID: \"711b7600-aba8-428e-aa8a-c661e05216d6\") " pod="openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r" Sep 11 14:57:31.121128 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:31.120952 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/711b7600-aba8-428e-aa8a-c661e05216d6-lib-modules\") pod \"perf-node-gather-daemonset-52l6r\" (UID: \"711b7600-aba8-428e-aa8a-c661e05216d6\") " pod="openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r" Sep 11 14:57:31.121128 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:31.121026 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/711b7600-aba8-428e-aa8a-c661e05216d6-proc\") pod \"perf-node-gather-daemonset-52l6r\" (UID: \"711b7600-aba8-428e-aa8a-c661e05216d6\") " pod="openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r" Sep 11 14:57:31.121128 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:31.121065 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/711b7600-aba8-428e-aa8a-c661e05216d6-podres\") pod \"perf-node-gather-daemonset-52l6r\" (UID: \"711b7600-aba8-428e-aa8a-c661e05216d6\") " pod="openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r" Sep 11 14:57:31.121312 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:31.121148 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/711b7600-aba8-428e-aa8a-c661e05216d6-sys\") pod \"perf-node-gather-daemonset-52l6r\" (UID: \"711b7600-aba8-428e-aa8a-c661e05216d6\") " pod="openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r" Sep 11 14:57:31.222725 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:31.222682 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6f7q\" (UniqueName: \"kubernetes.io/projected/711b7600-aba8-428e-aa8a-c661e05216d6-kube-api-access-p6f7q\") pod \"perf-node-gather-daemonset-52l6r\" (UID: \"711b7600-aba8-428e-aa8a-c661e05216d6\") " pod="openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r" Sep 11 14:57:31.222931 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:31.222752 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/711b7600-aba8-428e-aa8a-c661e05216d6-lib-modules\") pod \"perf-node-gather-daemonset-52l6r\" (UID: \"711b7600-aba8-428e-aa8a-c661e05216d6\") " pod="openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r" Sep 11 14:57:31.222931 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:31.222793 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/711b7600-aba8-428e-aa8a-c661e05216d6-proc\") pod \"perf-node-gather-daemonset-52l6r\" (UID: \"711b7600-aba8-428e-aa8a-c661e05216d6\") " pod="openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r" Sep 11 14:57:31.222931 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:31.222818 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/711b7600-aba8-428e-aa8a-c661e05216d6-podres\") pod \"perf-node-gather-daemonset-52l6r\" (UID: \"711b7600-aba8-428e-aa8a-c661e05216d6\") " pod="openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r" Sep 11 14:57:31.222931 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:31.222890 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/711b7600-aba8-428e-aa8a-c661e05216d6-sys\") pod \"perf-node-gather-daemonset-52l6r\" (UID: \"711b7600-aba8-428e-aa8a-c661e05216d6\") " pod="openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r" Sep 11 14:57:31.223170 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:31.222958 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/711b7600-aba8-428e-aa8a-c661e05216d6-lib-modules\") pod \"perf-node-gather-daemonset-52l6r\" (UID: \"711b7600-aba8-428e-aa8a-c661e05216d6\") " pod="openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r" Sep 11 14:57:31.223170 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:31.223013 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/711b7600-aba8-428e-aa8a-c661e05216d6-sys\") pod \"perf-node-gather-daemonset-52l6r\" (UID: \"711b7600-aba8-428e-aa8a-c661e05216d6\") " pod="openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r" Sep 11 14:57:31.223170 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:31.223031 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/711b7600-aba8-428e-aa8a-c661e05216d6-proc\") pod \"perf-node-gather-daemonset-52l6r\" (UID: \"711b7600-aba8-428e-aa8a-c661e05216d6\") " pod="openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r" Sep 11 14:57:31.223170 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:31.223113 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/711b7600-aba8-428e-aa8a-c661e05216d6-podres\") pod \"perf-node-gather-daemonset-52l6r\" (UID: \"711b7600-aba8-428e-aa8a-c661e05216d6\") " pod="openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r" Sep 11 14:57:31.233179 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:31.233147 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6f7q\" (UniqueName: \"kubernetes.io/projected/711b7600-aba8-428e-aa8a-c661e05216d6-kube-api-access-p6f7q\") pod \"perf-node-gather-daemonset-52l6r\" (UID: \"711b7600-aba8-428e-aa8a-c661e05216d6\") " pod="openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r" Sep 11 14:57:31.345118 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:31.345039 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r" Sep 11 14:57:31.621623 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:31.621264 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r"] Sep 11 14:57:31.626093 ip-10-0-135-17 kubenswrapper[2553]: W0911 14:57:31.625963 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod711b7600_aba8_428e_aa8a_c661e05216d6.slice/crio-e9f0fe541efb16beef28893adeca331afbde4660c1256057938687a10a6d4337 WatchSource:0}: Error finding container e9f0fe541efb16beef28893adeca331afbde4660c1256057938687a10a6d4337: Status 404 returned error can't find the container with id e9f0fe541efb16beef28893adeca331afbde4660c1256057938687a10a6d4337 Sep 11 14:57:32.341612 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:32.341572 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r" event={"ID":"711b7600-aba8-428e-aa8a-c661e05216d6","Type":"ContainerStarted","Data":"5078b3766da6363467581f3f001a65d66d2990b44e5a4f91fe54ff043110527c"} Sep 11 14:57:32.341823 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:32.341672 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r" event={"ID":"711b7600-aba8-428e-aa8a-c661e05216d6","Type":"ContainerStarted","Data":"e9f0fe541efb16beef28893adeca331afbde4660c1256057938687a10a6d4337"} Sep 11 14:57:32.342145 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:32.342098 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r" Sep 11 14:57:32.512724 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:32.512692 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-dvlrt_79a83328-5a3b-4302-a708-7c4d7152102b/volume-data-source-validator/0.log" Sep 11 14:57:33.713508 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:33.713477 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z5464_9e998861-da84-467d-a4e8-e959e24c336f/dns/0.log" Sep 11 14:57:33.740478 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:33.740450 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z5464_9e998861-da84-467d-a4e8-e959e24c336f/kube-rbac-proxy/0.log" Sep 11 14:57:33.851623 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:33.851593 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q25fm_7a24be49-54d5-46bf-b7ed-73b43b0bcb54/dns-node-resolver/0.log" Sep 11 14:57:34.516151 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:34.516123 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bn8wp_5159b394-744e-49c1-932d-a78367218c21/node-ca/0.log" Sep 11 14:57:36.274603 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:36.274574 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-769cf48dc6-74wxb_f40b578b-ce2a-4eca-96c9-0f88c9e90261/kube-auth-proxy/0.log" Sep 11 14:57:36.562159 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:36.562078 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-cvn67_35517420-4a51-4512-810b-834bb79de2f0/payload-processing/0.log" Sep 11 14:57:37.615576 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:37.615544 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l5fhb_3c28b757-a450-43a5-8c99-63a0e3824cc8/serve-healthcheck-canary/0.log" Sep 11 14:57:38.357512 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:38.357483 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r" Sep 11 14:57:38.387628 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:38.387571 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4bt5c/perf-node-gather-daemonset-52l6r" podStartSLOduration=7.387554746 podStartE2EDuration="7.387554746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:57:32.361580219 +0000 UTC m=+2562.975686564" watchObservedRunningTime="2026-09-11 14:57:38.387554746 +0000 UTC m=+2569.001661097" Sep 11 14:57:38.579709 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:38.579678 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-whsnc_af8a50bd-365d-4ed0-ae7a-7c45ef86224f/kube-rbac-proxy/0.log" Sep 11 14:57:38.621603 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:38.621533 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-whsnc_af8a50bd-365d-4ed0-ae7a-7c45ef86224f/exporter/0.log" Sep 11 14:57:38.651754 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:38.651722 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-whsnc_af8a50bd-365d-4ed0-ae7a-7c45ef86224f/extractor/0.log" Sep 11 14:57:41.096569 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:41.096532 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zsn9_b1166baf-c7df-4e63-9360-81dec348a479/konnectivity-agent/0.log" limit=1024 Sep 11 14:57:42.542610 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:42.542573 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-769cf48dc6-74wxb_f40b578b-ce2a-4eca-96c9-0f88c9e90261/kube-auth-proxy/0.log" Sep 11 14:57:42.724774 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:42.724687 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86d7f488b8-8xdsr_e018c54d-e91d-4b5d-997f-cf8884c3cac9/manager/0.log" Sep 11 14:57:42.784515 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:42.784475 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86d7f488b8-8xdsr_e018c54d-e91d-4b5d-997f-cf8884c3cac9/copy-manifests/0.log" Sep 11 14:57:42.800478 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:42.800446 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-cvn67_35517420-4a51-4512-810b-834bb79de2f0/payload-processing/0.log" Sep 11 14:57:44.271802 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:44.271765 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-55cb89464b-tncr8_69decb7e-f8a7-4e03-9add-0f25b3dbb59b/manager/0.log" Sep 11 14:57:44.970029 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:44.969987 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-769cf48dc6-74wxb_f40b578b-ce2a-4eca-96c9-0f88c9e90261/kube-auth-proxy/0.log" Sep 11 14:57:45.156563 ip-10-0-135-17 kubenswrapper[2553]: I0911 14:57:45.156535 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-cvn67_35517420-4a51-4512-810b-834bb79de2f0/payload-processing/0.log"